Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 13:59:36.288064 22750 test_util.cc:268] Using random seed: 57542444
I20240502 13:59:36.295727 22750 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:36.295837 22750 ts_itest-base.cc:113] --------------
I20240502 13:59:36.295890 22750 ts_itest-base.cc:114] 5 tablet servers
I20240502 13:59:36.295924 22750 ts_itest-base.cc:115] 3 replicas per TS
I20240502 13:59:36.295938 22750 ts_itest-base.cc:116] --------------
2024-05-02T13:59:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:36Z Disabled control of system clock
I20240502 13:59:36.305274 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:36305
--webserver_interface=127.22.55.190
--webserver_port=0
--builtin_ntp_servers=127.22.55.148:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.55.190:36305 with env {}
W20240502 13:59:36.405400 22762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:36.405592 22762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:36.405632 22762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:36.408591 22762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:36.408643 22762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:36.408663 22762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:36.408679 22762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:36.412178 22762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.55.190:36305
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.55.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:36.413034 22762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:36.414089 22762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.422103 22767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.422906 22762 server_base.cc:1034] running on GCE node
W20240502 13:59:36.423570 22768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.424541 22770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.424962 22762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:36.425768 22762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:36.426892 22762 hybrid_clock.cc:648] HybridClock initialized: now 1714658376426878 us; error 27 us; skew 500 ppm
I20240502 13:59:36.433851 22762 webserver.cc:460] Webserver started at http://127.22.55.190:43013/ using document root <none> and password file <none>
I20240502 13:59:36.434345 22762 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.434392 22762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.434532 22762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.436112 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b241dc4d98084235a75e7ca946f4f7fb"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.436522 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b241dc4d98084235a75e7ca946f4f7fb"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.439723 22762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:36.441627 22776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.442515 22762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:36.442615 22762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b241dc4d98084235a75e7ca946f4f7fb"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.442700 22762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.463747 22762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.464308 22762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:36.464452 22762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.470456 22762 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.190:36305
I20240502 13:59:36.470466 22828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.190:36305 every 8 connection(s)
I20240502 13:59:36.471357 22762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 13:59:36.473731 22829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:36.478302 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb: Bootstrap starting.
I20240502 13:59:36.480405 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 22762
I20240502 13:59:36.480430 22829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:36.480538 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 13:59:36.481062 22829 log.cc:824] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:36.482654 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb: No bootstrap required, opened a new log
I20240502 13:59:36.485453 22829 raft_consensus.cc:348] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b241dc4d98084235a75e7ca946f4f7fb" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 36305 } }
I20240502 13:59:36.485610 22829 raft_consensus.cc:374] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:36.485643 22829 raft_consensus.cc:729] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b241dc4d98084235a75e7ca946f4f7fb, State: Initialized, Role: FOLLOWER
I20240502 13:59:36.486025 22829 consensus_queue.cc:260] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b241dc4d98084235a75e7ca946f4f7fb" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 36305 } }
I20240502 13:59:36.486140 22829 raft_consensus.cc:388] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:36.486205 22829 raft_consensus.cc:482] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:36.486317 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:36.487219 22829 raft_consensus.cc:504] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b241dc4d98084235a75e7ca946f4f7fb" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 36305 } }
I20240502 13:59:36.487561 22829 leader_election.cc:302] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b241dc4d98084235a75e7ca946f4f7fb; no voters:
I20240502 13:59:36.487851 22829 leader_election.cc:288] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:36.487931 22834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:36.488086 22834 raft_consensus.cc:686] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [term 1 LEADER]: Becoming Leader. State: Replica: b241dc4d98084235a75e7ca946f4f7fb, State: Running, Role: LEADER
I20240502 13:59:36.488400 22834 consensus_queue.cc:237] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b241dc4d98084235a75e7ca946f4f7fb" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 36305 } }
I20240502 13:59:36.488911 22829 sys_catalog.cc:564] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:36.490172 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b241dc4d98084235a75e7ca946f4f7fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b241dc4d98084235a75e7ca946f4f7fb" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 36305 } } }
I20240502 13:59:36.490181 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b241dc4d98084235a75e7ca946f4f7fb. Latest consensus state: current_term: 1 leader_uuid: "b241dc4d98084235a75e7ca946f4f7fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b241dc4d98084235a75e7ca946f4f7fb" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 36305 } } }
I20240502 13:59:36.490376 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:36.490381 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:36.490779 22843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:36.493836 22843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:36.498788 22843 catalog_manager.cc:1347] Generated new cluster ID: cb8cb555078740ac954a3f6001da64e4
I20240502 13:59:36.498842 22843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:36.505190 22843 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:36.505937 22843 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:36.510792 22843 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b241dc4d98084235a75e7ca946f4f7fb: Generated new TSK 0
I20240502 13:59:36.511364 22843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:36.520749 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.129:0
--local_ip_for_outbound_sockets=127.22.55.129
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--builtin_ntp_servers=127.22.55.148:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 13:59:36.625047 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:36.625273 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:36.625321 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:36.628314 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:36.628410 22853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.129
I20240502 13:59:36.631925 22853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--never_fsync=true
--heap_profile_path=/tmp/kudu.22853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.129
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:36.632751 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:36.633821 22853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.642581 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.643393 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.644166 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.644905 22853 server_base.cc:1034] running on GCE node
I20240502 13:59:36.645252 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:36.645795 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:36.646896 22853 hybrid_clock.cc:648] HybridClock initialized: now 1714658376646885 us; error 26 us; skew 500 ppm
I20240502 13:59:36.648604 22853 webserver.cc:460] Webserver started at http://127.22.55.129:35123/ using document root <none> and password file <none>
I20240502 13:59:36.649116 22853 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.649169 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.649348 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.650991 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "82c33d0b68e64adea5368f1881dc62f7"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.651435 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "82c33d0b68e64adea5368f1881dc62f7"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.654615 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:36.656657 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.657555 22853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:36.657655 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "82c33d0b68e64adea5368f1881dc62f7"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.657727 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.668089 22853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.668646 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:36.668785 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.669242 22853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:36.670091 22853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:36.670140 22853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.670194 22853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:36.670219 22853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.678277 22853 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.129:36049
I20240502 13:59:36.678301 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.129:36049 every 8 connection(s)
I20240502 13:59:36.679198 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 13:59:36.686111 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 22853
I20240502 13:59:36.686224 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 13:59:36.687299 22981 heartbeater.cc:347] Connected to a master server at 127.22.55.190:36305
I20240502 13:59:36.687530 22981 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.688071 22981 heartbeater.cc:511] Master 127.22.55.190:36305 requested a full tablet report, sending...
I20240502 13:59:36.688309 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.130:0
--local_ip_for_outbound_sockets=127.22.55.130
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--builtin_ntp_servers=127.22.55.148:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 13:59:36.689489 22793 ts_manager.cc:194] Registered new tserver with Master: 82c33d0b68e64adea5368f1881dc62f7 (127.22.55.129:36049)
I20240502 13:59:36.690968 22793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.129:39757
W20240502 13:59:36.792078 22984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:36.792289 22984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:36.792328 22984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:36.795225 22984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:36.795313 22984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.130
I20240502 13:59:36.798707 22984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--never_fsync=true
--heap_profile_path=/tmp/kudu.22984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.130
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:36.799505 22984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:36.800531 22984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.808663 22992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.809191 22990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.809850 22989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.810921 22984 server_base.cc:1034] running on GCE node
I20240502 13:59:36.811286 22984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:36.811785 22984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:36.812942 22984 hybrid_clock.cc:648] HybridClock initialized: now 1714658376812931 us; error 23 us; skew 500 ppm
I20240502 13:59:36.814757 22984 webserver.cc:460] Webserver started at http://127.22.55.130:34249/ using document root <none> and password file <none>
I20240502 13:59:36.815304 22984 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.815372 22984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.815533 22984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.817001 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "51edae149b114cb2a0fc6bff9a75b99d"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.817458 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "51edae149b114cb2a0fc6bff9a75b99d"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.820564 22984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:36.822455 22998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.823356 22984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:36.823453 22984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "51edae149b114cb2a0fc6bff9a75b99d"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.823525 22984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:36.843068 22984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:36.843631 22984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:36.843780 22984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:36.844277 22984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:36.845064 22984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:36.845116 22984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.845168 22984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:36.845191 22984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.853195 22984 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.130:35779
I20240502 13:59:36.853228 23111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.130:35779 every 8 connection(s)
I20240502 13:59:36.854027 22984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 13:59:36.855577 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 22984
I20240502 13:59:36.855682 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 13:59:36.857632 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.131:0
--local_ip_for_outbound_sockets=127.22.55.131
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--builtin_ntp_servers=127.22.55.148:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 13:59:36.862376 23112 heartbeater.cc:347] Connected to a master server at 127.22.55.190:36305
I20240502 13:59:36.862571 23112 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:36.863058 23112 heartbeater.cc:511] Master 127.22.55.190:36305 requested a full tablet report, sending...
I20240502 13:59:36.863991 22793 ts_manager.cc:194] Registered new tserver with Master: 51edae149b114cb2a0fc6bff9a75b99d (127.22.55.130:35779)
I20240502 13:59:36.864784 22793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.130:59807
W20240502 13:59:36.959569 23115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:36.959761 23115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:36.959801 23115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:36.962724 23115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:36.962818 23115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.131
I20240502 13:59:36.966238 23115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--never_fsync=true
--heap_profile_path=/tmp/kudu.23115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.131
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:36.967033 23115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:36.968062 23115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:36.976231 23123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.976841 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:36.977705 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:36.978574 23115 server_base.cc:1034] running on GCE node
I20240502 13:59:36.978927 23115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:36.979427 23115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:36.980576 23115 hybrid_clock.cc:648] HybridClock initialized: now 1714658376980556 us; error 30 us; skew 500 ppm
I20240502 13:59:36.982322 23115 webserver.cc:460] Webserver started at http://127.22.55.131:35089/ using document root <none> and password file <none>
I20240502 13:59:36.982861 23115 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:36.982942 23115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:36.983129 23115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:36.984647 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e87f3cb14a1a4982909c5a526a69a081"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.985087 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e87f3cb14a1a4982909c5a526a69a081"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.988155 23115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 13:59:36.990154 23129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:36.991091 23115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:36.991216 23115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e87f3cb14a1a4982909c5a526a69a081"
format_stamp: "Formatted at 2024-05-02 13:59:36 on dist-test-slave-sdf5"
I20240502 13:59:36.991289 23115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:37.005671 23115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:37.006217 23115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:37.006338 23115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:37.006781 23115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:37.007666 23115 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:37.007710 23115 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.007745 23115 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:37.007766 23115 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.015630 23115 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.131:43241
I20240502 13:59:37.015663 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.131:43241 every 8 connection(s)
I20240502 13:59:37.016460 23115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:37.022933 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 23115
I20240502 13:59:37.023029 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 13:59:37.024472 23243 heartbeater.cc:347] Connected to a master server at 127.22.55.190:36305
I20240502 13:59:37.024685 23243 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:37.025028 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.132:0
--local_ip_for_outbound_sockets=127.22.55.132
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--builtin_ntp_servers=127.22.55.148:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 13:59:37.025274 23243 heartbeater.cc:511] Master 127.22.55.190:36305 requested a full tablet report, sending...
I20240502 13:59:37.026279 22793 ts_manager.cc:194] Registered new tserver with Master: e87f3cb14a1a4982909c5a526a69a081 (127.22.55.131:43241)
I20240502 13:59:37.026911 22793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.131:55529
W20240502 13:59:37.129427 23246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:37.129629 23246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:37.129701 23246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:37.132651 23246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:37.132803 23246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.132
I20240502 13:59:37.136271 23246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--never_fsync=true
--heap_profile_path=/tmp/kudu.23246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.132
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:37.137353 23246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:37.138772 23246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:37.146831 23254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:37.147538 23252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:37.148152 23251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:37.149209 23246 server_base.cc:1034] running on GCE node
I20240502 13:59:37.149525 23246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:37.150020 23246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:37.151154 23246 hybrid_clock.cc:648] HybridClock initialized: now 1714658377151143 us; error 24 us; skew 500 ppm
I20240502 13:59:37.152751 23246 webserver.cc:460] Webserver started at http://127.22.55.132:37559/ using document root <none> and password file <none>
I20240502 13:59:37.153244 23246 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:37.153295 23246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:37.153462 23246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:37.155170 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b7559f66066e42769568fd3256ca1313"
format_stamp: "Formatted at 2024-05-02 13:59:37 on dist-test-slave-sdf5"
I20240502 13:59:37.155634 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b7559f66066e42769568fd3256ca1313"
format_stamp: "Formatted at 2024-05-02 13:59:37 on dist-test-slave-sdf5"
I20240502 13:59:37.158638 23246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:37.160686 23260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.161607 23246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:37.161702 23246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b7559f66066e42769568fd3256ca1313"
format_stamp: "Formatted at 2024-05-02 13:59:37 on dist-test-slave-sdf5"
I20240502 13:59:37.161777 23246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:37.172652 23246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:37.173206 23246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:37.173367 23246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:37.173842 23246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:37.174702 23246 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:37.174749 23246 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.174821 23246 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:37.174875 23246 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.182688 23246 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.132:41417
I20240502 13:59:37.182727 23373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.132:41417 every 8 connection(s)
I20240502 13:59:37.183530 23246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:37.190424 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 23246
I20240502 13:59:37.190550 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 13:59:37.191460 23374 heartbeater.cc:347] Connected to a master server at 127.22.55.190:36305
I20240502 13:59:37.191660 23374 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:37.192196 23374 heartbeater.cc:511] Master 127.22.55.190:36305 requested a full tablet report, sending...
I20240502 13:59:37.192544 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.133:0
--local_ip_for_outbound_sockets=127.22.55.133
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--builtin_ntp_servers=127.22.55.148:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 13:59:37.193017 22793 ts_manager.cc:194] Registered new tserver with Master: b7559f66066e42769568fd3256ca1313 (127.22.55.132:41417)
I20240502 13:59:37.193636 22793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.132:58777
W20240502 13:59:37.299315 23377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:37.299523 23377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:37.299568 23377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:37.302536 23377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:37.302659 23377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.133
I20240502 13:59:37.306049 23377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:36305
--never_fsync=true
--heap_profile_path=/tmp/kudu.23377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.133
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:37.306921 23377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:37.308041 23377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:37.316638 23382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:37.317384 23385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:37.318295 23383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:37.319027 23377 server_base.cc:1034] running on GCE node
I20240502 13:59:37.319344 23377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:37.319833 23377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:37.320976 23377 hybrid_clock.cc:648] HybridClock initialized: now 1714658377320964 us; error 26 us; skew 500 ppm
I20240502 13:59:37.322700 23377 webserver.cc:460] Webserver started at http://127.22.55.133:40521/ using document root <none> and password file <none>
I20240502 13:59:37.323249 23377 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:37.323308 23377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:37.323509 23377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:37.325071 23377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5bad99be36894fd4bcd7f981af3f5a93"
format_stamp: "Formatted at 2024-05-02 13:59:37 on dist-test-slave-sdf5"
I20240502 13:59:37.325524 23377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5bad99be36894fd4bcd7f981af3f5a93"
format_stamp: "Formatted at 2024-05-02 13:59:37 on dist-test-slave-sdf5"
I20240502 13:59:37.328773 23377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:37.330719 23391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.331676 23377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:37.331781 23377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5bad99be36894fd4bcd7f981af3f5a93"
format_stamp: "Formatted at 2024-05-02 13:59:37 on dist-test-slave-sdf5"
I20240502 13:59:37.331856 23377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:37.356281 23377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:37.356930 23377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:37.357081 23377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:37.357599 23377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:37.358481 23377 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:37.358527 23377 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.358589 23377 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:37.358615 23377 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:37.366776 23377 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.133:35293
I20240502 13:59:37.366803 23504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.133:35293 every 8 connection(s)
I20240502 13:59:37.367666 23377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:37.368542 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 23377
I20240502 13:59:37.368628 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 13:59:37.375767 23505 heartbeater.cc:347] Connected to a master server at 127.22.55.190:36305
I20240502 13:59:37.375984 23505 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:37.376463 23505 heartbeater.cc:511] Master 127.22.55.190:36305 requested a full tablet report, sending...
I20240502 13:59:37.377322 22793 ts_manager.cc:194] Registered new tserver with Master: 5bad99be36894fd4bcd7f981af3f5a93 (127.22.55.133:35293)
I20240502 13:59:37.377837 22793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.133:38391
I20240502 13:59:37.382701 22750 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 13:59:37.393318 22793 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45026:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 13:59:37.410440 23177 tablet_service.cc:1467] Processing CreateTablet for tablet f5fd17545fae4c5a9edd37508d450417 (DEFAULT_TABLE table=TestTable [id=2c176ee8f1da489796539271c08a579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:37.410455 23439 tablet_service.cc:1467] Processing CreateTablet for tablet f5fd17545fae4c5a9edd37508d450417 (DEFAULT_TABLE table=TestTable [id=2c176ee8f1da489796539271c08a579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:37.411331 23439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fd17545fae4c5a9edd37508d450417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:37.411345 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fd17545fae4c5a9edd37508d450417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:37.412461 22915 tablet_service.cc:1467] Processing CreateTablet for tablet f5fd17545fae4c5a9edd37508d450417 (DEFAULT_TABLE table=TestTable [id=2c176ee8f1da489796539271c08a579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:37.413252 22915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fd17545fae4c5a9edd37508d450417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:37.416010 23522 tablet_bootstrap.cc:492] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93: Bootstrap starting.
I20240502 13:59:37.417093 23523 tablet_bootstrap.cc:492] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081: Bootstrap starting.
I20240502 13:59:37.418151 23524 tablet_bootstrap.cc:492] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7: Bootstrap starting.
I20240502 13:59:37.418861 23522 tablet_bootstrap.cc:654] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:37.419188 23523 tablet_bootstrap.cc:654] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:37.419688 23522 log.cc:824] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:37.419935 23523 log.cc:824] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:37.420044 23524 tablet_bootstrap.cc:654] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:37.420717 23524 log.cc:824] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:37.421346 23522 tablet_bootstrap.cc:492] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93: No bootstrap required, opened a new log
I20240502 13:59:37.421420 23523 tablet_bootstrap.cc:492] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081: No bootstrap required, opened a new log
I20240502 13:59:37.421519 23522 ts_tablet_manager.cc:1367] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 13:59:37.421587 23523 ts_tablet_manager.cc:1367] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:37.422055 23524 tablet_bootstrap.cc:492] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7: No bootstrap required, opened a new log
I20240502 13:59:37.422212 23524 ts_tablet_manager.cc:1367] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 13:59:37.424731 23524 raft_consensus.cc:348] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.424916 23524 raft_consensus.cc:374] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:37.424995 23524 raft_consensus.cc:729] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c33d0b68e64adea5368f1881dc62f7, State: Initialized, Role: FOLLOWER
I20240502 13:59:37.425251 23522 raft_consensus.cc:348] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.425251 23523 raft_consensus.cc:348] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.425436 23522 raft_consensus.cc:374] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:37.425436 23523 raft_consensus.cc:374] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:37.425410 23524 consensus_queue.cc:260] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.425491 23523 raft_consensus.cc:729] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87f3cb14a1a4982909c5a526a69a081, State: Initialized, Role: FOLLOWER
I20240502 13:59:37.425491 23522 raft_consensus.cc:729] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bad99be36894fd4bcd7f981af3f5a93, State: Initialized, Role: FOLLOWER
I20240502 13:59:37.425905 23522 consensus_queue.cc:260] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.425905 23523 consensus_queue.cc:260] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.426678 23522 ts_tablet_manager.cc:1398] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 13:59:37.426687 23524 ts_tablet_manager.cc:1398] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 13:59:37.426719 23523 ts_tablet_manager.cc:1398] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 13:59:37.427042 22981 heartbeater.cc:503] Master 127.22.55.190:36305 was elected leader, sending a full tablet report...
I20240502 13:59:37.427060 23505 heartbeater.cc:503] Master 127.22.55.190:36305 was elected leader, sending a full tablet report...
I20240502 13:59:37.427060 23243 heartbeater.cc:503] Master 127.22.55.190:36305 was elected leader, sending a full tablet report...
W20240502 13:59:37.430462 22982 tablet.cc:2369] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:37.517504 23244 tablet.cc:2369] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:37.598109 23528 raft_consensus.cc:482] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:37.598323 23528 raft_consensus.cc:504] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.599399 23528 leader_election.cc:288] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e87f3cb14a1a4982909c5a526a69a081 (127.22.55.131:43241), 5bad99be36894fd4bcd7f981af3f5a93 (127.22.55.133:35293)
I20240502 13:59:37.603473 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fd17545fae4c5a9edd37508d450417" candidate_uuid: "82c33d0b68e64adea5368f1881dc62f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87f3cb14a1a4982909c5a526a69a081" is_pre_election: true
I20240502 13:59:37.603566 23459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fd17545fae4c5a9edd37508d450417" candidate_uuid: "82c33d0b68e64adea5368f1881dc62f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bad99be36894fd4bcd7f981af3f5a93" is_pre_election: true
I20240502 13:59:37.603737 23197 raft_consensus.cc:2454] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c33d0b68e64adea5368f1881dc62f7 in term 0.
I20240502 13:59:37.603830 23459 raft_consensus.cc:2454] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c33d0b68e64adea5368f1881dc62f7 in term 0.
I20240502 13:59:37.604158 22869 leader_election.cc:302] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c33d0b68e64adea5368f1881dc62f7, e87f3cb14a1a4982909c5a526a69a081; no voters:
I20240502 13:59:37.604363 23528 raft_consensus.cc:2781] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:37.604435 23528 raft_consensus.cc:482] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:37.604490 23528 raft_consensus.cc:3037] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:37.605397 23528 raft_consensus.cc:504] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.605683 23528 leader_election.cc:288] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [CANDIDATE]: Term 1 election: Requested vote from peers e87f3cb14a1a4982909c5a526a69a081 (127.22.55.131:43241), 5bad99be36894fd4bcd7f981af3f5a93 (127.22.55.133:35293)
I20240502 13:59:37.605973 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fd17545fae4c5a9edd37508d450417" candidate_uuid: "82c33d0b68e64adea5368f1881dc62f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87f3cb14a1a4982909c5a526a69a081"
I20240502 13:59:37.606046 23459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fd17545fae4c5a9edd37508d450417" candidate_uuid: "82c33d0b68e64adea5368f1881dc62f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bad99be36894fd4bcd7f981af3f5a93"
I20240502 13:59:37.606073 23197 raft_consensus.cc:3037] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:37.606154 23459 raft_consensus.cc:3037] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:37.606945 23197 raft_consensus.cc:2454] T f5fd17545fae4c5a9edd37508d450417 P e87f3cb14a1a4982909c5a526a69a081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c33d0b68e64adea5368f1881dc62f7 in term 1.
I20240502 13:59:37.607112 23459 raft_consensus.cc:2454] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c33d0b68e64adea5368f1881dc62f7 in term 1.
I20240502 13:59:37.607213 22869 leader_election.cc:302] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c33d0b68e64adea5368f1881dc62f7, e87f3cb14a1a4982909c5a526a69a081; no voters:
I20240502 13:59:37.607379 23528 raft_consensus.cc:2781] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:37.607563 23528 raft_consensus.cc:686] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [term 1 LEADER]: Becoming Leader. State: Replica: 82c33d0b68e64adea5368f1881dc62f7, State: Running, Role: LEADER
I20240502 13:59:37.607841 23528 consensus_queue.cc:237] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } }
I20240502 13:59:37.610634 22791 catalog_manager.cc:5484] T f5fd17545fae4c5a9edd37508d450417 P 82c33d0b68e64adea5368f1881dc62f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c33d0b68e64adea5368f1881dc62f7 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "82c33d0b68e64adea5368f1881dc62f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87f3cb14a1a4982909c5a526a69a081" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 43241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c33d0b68e64adea5368f1881dc62f7" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bad99be36894fd4bcd7f981af3f5a93" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 35293 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:37.618523 23506 tablet.cc:2369] T f5fd17545fae4c5a9edd37508d450417 P 5bad99be36894fd4bcd7f981af3f5a93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:37.637552 22750 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 13:59:37.639061 22750 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 82c33d0b68e64adea5368f1881dc62f7 to finish bootstrapping
I20240502 13:59:37.643947 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 51edae149b114cb2a0fc6bff9a75b99d to finish bootstrapping
I20240502 13:59:37.649509 22750 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e87f3cb14a1a4982909c5a526a69a081 to finish bootstrapping
I20240502 13:59:37.653890 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b7559f66066e42769568fd3256ca1313 to finish bootstrapping
I20240502 13:59:37.658896 22750 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5bad99be36894fd4bcd7f981af3f5a93 to finish bootstrapping
I20240502 13:59:37.837555 23046 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.838394 22915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:37.838471 23439 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.839198 23308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.845718 23177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.866447 23112 heartbeater.cc:503] Master 127.22.55.190:36305 was elected leader, sending a full tablet report...
I20240502 13:59:37.926249 23046 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.927603 23308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.928100 23439 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.929770 22915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:37.935078 23177 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:37.966475 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 22853
I20240502 13:59:37.974857 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 22984
I20240502 13:59:37.980551 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 23115
I20240502 13:59:37.985850 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 23246
I20240502 13:59:37.991214 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 23377
I20240502 13:59:37.996395 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 22762
2024-05-02T13:59:37Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (1732 ms)
[----------] 1 test from AdminCliTest (1732 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240502 13:59:38.018831 22750 test_util.cc:268] Using random seed: 59273226
I20240502 13:59:38.019572 22750 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:38.019613 22750 ts_itest-base.cc:113] --------------
I20240502 13:59:38.019624 22750 ts_itest-base.cc:114] 5 tablet servers
I20240502 13:59:38.019634 22750 ts_itest-base.cc:115] 3 replicas per TS
I20240502 13:59:38.019658 22750 ts_itest-base.cc:116] --------------
2024-05-02T13:59:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:38Z Disabled control of system clock
I20240502 13:59:38.027179 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:40859
--webserver_interface=127.22.55.190
--webserver_port=0
--builtin_ntp_servers=127.22.55.148:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.55.190:40859
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 13:59:38.130970 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.131182 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.131248 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.134099 23612 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:38.134171 23612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:38.134215 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.134253 23612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:38.134294 23612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:38.137754 23612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.55.190:40859
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:40859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.55.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.138541 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.139693 23612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.147287 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.148111 23618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.148766 23620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.149482 23612 server_base.cc:1034] running on GCE node
I20240502 13:59:38.149904 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.150694 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.151850 23612 hybrid_clock.cc:648] HybridClock initialized: now 1714658378151847 us; error 44 us; skew 500 ppm
I20240502 13:59:38.153613 23612 webserver.cc:460] Webserver started at http://127.22.55.190:41153/ using document root <none> and password file <none>
I20240502 13:59:38.154130 23612 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.154186 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.154345 23612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.155964 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0a0c733265114e4d8a3f689f540f468d"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.156409 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0a0c733265114e4d8a3f689f540f468d"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.159557 23612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:38.161574 23626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.162473 23612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:38.162572 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0a0c733265114e4d8a3f689f540f468d"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.162647 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.175417 23612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.176003 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.176157 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.182441 23612 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.190:40859
I20240502 13:59:38.182448 23678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.190:40859 every 8 connection(s)
I20240502 13:59:38.183396 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 13:59:38.185812 23679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:38.190606 23679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d: Bootstrap starting.
I20240502 13:59:38.191506 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 23612
I20240502 13:59:38.191639 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 13:59:38.193145 23679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:38.193773 23679 log.cc:824] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:38.195520 23679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d: No bootstrap required, opened a new log
I20240502 13:59:38.198341 23679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0c733265114e4d8a3f689f540f468d" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 40859 } }
I20240502 13:59:38.198539 23679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:38.198607 23679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0c733265114e4d8a3f689f540f468d, State: Initialized, Role: FOLLOWER
I20240502 13:59:38.199054 23679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0c733265114e4d8a3f689f540f468d" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 40859 } }
I20240502 13:59:38.199199 23679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:38.199259 23679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:38.199375 23679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:38.200004 23679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0c733265114e4d8a3f689f540f468d" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 40859 } }
I20240502 13:59:38.200343 23679 leader_election.cc:302] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a0c733265114e4d8a3f689f540f468d; no voters:
I20240502 13:59:38.200562 23679 leader_election.cc:288] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:38.200660 23684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:38.201164 23684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [term 1 LEADER]: Becoming Leader. State: Replica: 0a0c733265114e4d8a3f689f540f468d, State: Running, Role: LEADER
I20240502 13:59:38.201419 23684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0c733265114e4d8a3f689f540f468d" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 40859 } }
I20240502 13:59:38.201503 23679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:38.202903 23686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0c733265114e4d8a3f689f540f468d. Latest consensus state: current_term: 1 leader_uuid: "0a0c733265114e4d8a3f689f540f468d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0c733265114e4d8a3f689f540f468d" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 40859 } } }
I20240502 13:59:38.202944 23685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a0c733265114e4d8a3f689f540f468d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0c733265114e4d8a3f689f540f468d" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 40859 } } }
I20240502 13:59:38.203009 23686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:38.203039 23685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d [sys.catalog]: This master's current role is: LEADER
W20240502 13:59:38.205806 23699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:38.205863 23699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:38.205925 23694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:38.206712 23694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:38.211269 23694 catalog_manager.cc:1347] Generated new cluster ID: 88e114c6a6be4c949088543bde6f8794
I20240502 13:59:38.211331 23694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:38.225324 23694 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:38.226158 23694 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:38.232677 23694 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0a0c733265114e4d8a3f689f540f468d: Generated new TSK 0
I20240502 13:59:38.233289 23694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:38.242269 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.129:0
--local_ip_for_outbound_sockets=127.22.55.129
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--builtin_ntp_servers=127.22.55.148:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 13:59:38.344255 23703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.344478 23703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.344544 23703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.347559 23703 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:38.347626 23703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.347714 23703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.129
I20240502 13:59:38.351200 23703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--never_fsync=true
--heap_profile_path=/tmp/kudu.23703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.129
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.352032 23703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.353062 23703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.361707 23708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.362287 23709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.362970 23711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.364054 23703 server_base.cc:1034] running on GCE node
I20240502 13:59:38.364404 23703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.364895 23703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.366039 23703 hybrid_clock.cc:648] HybridClock initialized: now 1714658378366020 us; error 31 us; skew 500 ppm
I20240502 13:59:38.367966 23703 webserver.cc:460] Webserver started at http://127.22.55.129:37053/ using document root <none> and password file <none>
I20240502 13:59:38.368510 23703 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.368566 23703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.368736 23703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.370489 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c3538f5fb7584c91b2a8d0ef82163780"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.371008 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c3538f5fb7584c91b2a8d0ef82163780"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.374361 23703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:38.376482 23717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.377434 23703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:38.377529 23703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c3538f5fb7584c91b2a8d0ef82163780"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.377607 23703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.397852 23703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.398507 23703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.398674 23703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.399276 23703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.400313 23703 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.400373 23703 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.400447 23703 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.400480 23703 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.409955 23703 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.129:45565
I20240502 13:59:38.409992 23830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.129:45565 every 8 connection(s)
I20240502 13:59:38.410929 23703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 13:59:38.418015 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 23703
I20240502 13:59:38.418135 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 13:59:38.420068 23831 heartbeater.cc:347] Connected to a master server at 127.22.55.190:40859
I20240502 13:59:38.420286 23831 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.420526 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.130:0
--local_ip_for_outbound_sockets=127.22.55.130
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--builtin_ntp_servers=127.22.55.148:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:38.420778 23831 heartbeater.cc:511] Master 127.22.55.190:40859 requested a full tablet report, sending...
I20240502 13:59:38.422287 23643 ts_manager.cc:194] Registered new tserver with Master: c3538f5fb7584c91b2a8d0ef82163780 (127.22.55.129:45565)
I20240502 13:59:38.423457 23643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.129:57769
W20240502 13:59:38.528980 23834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.529248 23834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.529327 23834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.532500 23834 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:38.532578 23834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.532680 23834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.130
I20240502 13:59:38.536283 23834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--never_fsync=true
--heap_profile_path=/tmp/kudu.23834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.130
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.537202 23834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.538321 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.546903 23839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.547544 23842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.548234 23840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.549338 23834 server_base.cc:1034] running on GCE node
I20240502 13:59:38.549724 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.550261 23834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.551476 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714658378551451 us; error 40 us; skew 500 ppm
I20240502 13:59:38.553422 23834 webserver.cc:460] Webserver started at http://127.22.55.130:38309/ using document root <none> and password file <none>
I20240502 13:59:38.553961 23834 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.554023 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.554198 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.555886 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "00b6f6c967f5452cbbf470dc26af4d4c"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.556363 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "00b6f6c967f5452cbbf470dc26af4d4c"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.559791 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:38.561940 23848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.562919 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:38.563027 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "00b6f6c967f5452cbbf470dc26af4d4c"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.563112 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.578681 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.579357 23834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.579526 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.580044 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.580951 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.581001 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.581063 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.581097 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.589633 23834 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.130:39157
I20240502 13:59:38.589655 23961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.130:39157 every 8 connection(s)
I20240502 13:59:38.590636 23834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 13:59:38.596280 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 23834
I20240502 13:59:38.596403 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 13:59:38.598727 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.131:0
--local_ip_for_outbound_sockets=127.22.55.131
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--builtin_ntp_servers=127.22.55.148:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:38.599771 23962 heartbeater.cc:347] Connected to a master server at 127.22.55.190:40859
I20240502 13:59:38.600016 23962 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.600648 23962 heartbeater.cc:511] Master 127.22.55.190:40859 requested a full tablet report, sending...
I20240502 13:59:38.601701 23643 ts_manager.cc:194] Registered new tserver with Master: 00b6f6c967f5452cbbf470dc26af4d4c (127.22.55.130:39157)
I20240502 13:59:38.602624 23643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.130:44017
W20240502 13:59:38.706218 23965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.706434 23965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.706492 23965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.709412 23965 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:38.709476 23965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.709540 23965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.131
I20240502 13:59:38.713073 23965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--never_fsync=true
--heap_profile_path=/tmp/kudu.23965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.131
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.713917 23965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.715080 23965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.723462 23973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.724273 23970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.725006 23971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.725725 23965 server_base.cc:1034] running on GCE node
I20240502 13:59:38.726085 23965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.726639 23965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.727787 23965 hybrid_clock.cc:648] HybridClock initialized: now 1714658378727777 us; error 33 us; skew 500 ppm
I20240502 13:59:38.729717 23965 webserver.cc:460] Webserver started at http://127.22.55.131:38209/ using document root <none> and password file <none>
I20240502 13:59:38.730300 23965 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.730362 23965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.730562 23965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.732259 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "44f704db56294d6fae53be7563d4b07b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.732746 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "44f704db56294d6fae53be7563d4b07b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.736081 23965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:38.738157 23979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.739187 23965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:38.739300 23965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "44f704db56294d6fae53be7563d4b07b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.739387 23965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.755760 23965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.756340 23965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.756502 23965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.757004 23965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.757869 23965 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.757918 23965 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.757978 23965 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.758006 23965 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.766091 23965 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.131:38469
I20240502 13:59:38.766208 24092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.131:38469 every 8 connection(s)
I20240502 13:59:38.767105 23965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:38.775032 24093 heartbeater.cc:347] Connected to a master server at 127.22.55.190:40859
I20240502 13:59:38.775090 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 23965
I20240502 13:59:38.775218 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 13:59:38.775259 24093 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.775677 24093 heartbeater.cc:511] Master 127.22.55.190:40859 requested a full tablet report, sending...
I20240502 13:59:38.776600 23642 ts_manager.cc:194] Registered new tserver with Master: 44f704db56294d6fae53be7563d4b07b (127.22.55.131:38469)
I20240502 13:59:38.777158 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.132:0
--local_ip_for_outbound_sockets=127.22.55.132
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--builtin_ntp_servers=127.22.55.148:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:38.777164 23642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.131:47623
W20240502 13:59:38.879992 24096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.880225 24096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.880306 24096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.883464 24096 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:38.883535 24096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.883622 24096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.132
I20240502 13:59:38.887113 24096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--never_fsync=true
--heap_profile_path=/tmp/kudu.24096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.132
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.887979 24096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.889086 24096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.897625 24101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.898514 24104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.899221 24102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.899957 24096 server_base.cc:1034] running on GCE node
I20240502 13:59:38.900308 24096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.900815 24096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.901978 24096 hybrid_clock.cc:648] HybridClock initialized: now 1714658378901972 us; error 27 us; skew 500 ppm
I20240502 13:59:38.903793 24096 webserver.cc:460] Webserver started at http://127.22.55.132:43105/ using document root <none> and password file <none>
I20240502 13:59:38.904326 24096 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.904390 24096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.904577 24096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.906131 24096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3ec7c1e7b3ed439bb678fe4df426f427"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.906580 24096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3ec7c1e7b3ed439bb678fe4df426f427"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.909781 24096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:38.913300 24110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.914235 24096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:38.914337 24096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3ec7c1e7b3ed439bb678fe4df426f427"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-sdf5"
I20240502 13:59:38.914425 24096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.934265 24096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.934940 24096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.935099 24096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.935618 24096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.936479 24096 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.936525 24096 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.936570 24096 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.936602 24096 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.944614 24096 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.132:40247
I20240502 13:59:38.944640 24223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.132:40247 every 8 connection(s)
I20240502 13:59:38.945497 24096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:38.952898 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 24096
I20240502 13:59:38.953009 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 13:59:38.953595 24224 heartbeater.cc:347] Connected to a master server at 127.22.55.190:40859
I20240502 13:59:38.953825 24224 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.954263 24224 heartbeater.cc:511] Master 127.22.55.190:40859 requested a full tablet report, sending...
I20240502 13:59:38.955266 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.133:0
--local_ip_for_outbound_sockets=127.22.55.133
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--builtin_ntp_servers=127.22.55.148:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:38.955300 23642 ts_manager.cc:194] Registered new tserver with Master: 3ec7c1e7b3ed439bb678fe4df426f427 (127.22.55.132:40247)
I20240502 13:59:38.956043 23642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.132:57471
W20240502 13:59:39.060603 24227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.060815 24227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.060870 24227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.063958 24227 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:39.064023 24227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.064124 24227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.133
I20240502 13:59:39.067739 24227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:40859
--never_fsync=true
--heap_profile_path=/tmp/kudu.24227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.133
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.068653 24227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.069792 24227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.078646 24233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.079140 24235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.080220 24232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.081017 24227 server_base.cc:1034] running on GCE node
I20240502 13:59:39.081421 24227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.081965 24227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.083251 24227 hybrid_clock.cc:648] HybridClock initialized: now 1714658379083238 us; error 27 us; skew 500 ppm
I20240502 13:59:39.085183 24227 webserver.cc:460] Webserver started at http://127.22.55.133:45007/ using document root <none> and password file <none>
I20240502 13:59:39.085738 24227 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.085798 24227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.085978 24227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.087654 24227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-sdf5"
I20240502 13:59:39.088105 24227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-sdf5"
I20240502 13:59:39.091394 24227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:39.093560 24241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.094760 24227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:39.094940 24227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-sdf5"
I20240502 13:59:39.095058 24227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.119345 24227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.119921 24227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.120054 24227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.120533 24227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.121363 24227 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.121408 24227 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.121448 24227 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.121469 24227 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.129714 24227 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.133:37699
I20240502 13:59:39.129745 24354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.133:37699 every 8 connection(s)
I20240502 13:59:39.130613 24227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:39.131336 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 24227
I20240502 13:59:39.131446 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 13:59:39.139142 24355 heartbeater.cc:347] Connected to a master server at 127.22.55.190:40859
I20240502 13:59:39.139319 24355 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.139683 24355 heartbeater.cc:511] Master 127.22.55.190:40859 requested a full tablet report, sending...
I20240502 13:59:39.140477 23642 ts_manager.cc:194] Registered new tserver with Master: 8eb2cb5bf7d34198bb2287df4eeaba3a (127.22.55.133:37699)
I20240502 13:59:39.140969 23642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.133:46901
I20240502 13:59:39.146098 22750 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 13:59:39.153857 23642 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43924:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 13:59:39.171805 24289 tablet_service.cc:1467] Processing CreateTablet for tablet c5ed877410cc46fb8ab71a39bc9a39f8 (DEFAULT_TABLE table=TestTable [id=96744a1cdf364f2292e5ad0befa9d646]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:39.171809 24027 tablet_service.cc:1467] Processing CreateTablet for tablet c5ed877410cc46fb8ab71a39bc9a39f8 (DEFAULT_TABLE table=TestTable [id=96744a1cdf364f2292e5ad0befa9d646]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:39.171809 23896 tablet_service.cc:1467] Processing CreateTablet for tablet c5ed877410cc46fb8ab71a39bc9a39f8 (DEFAULT_TABLE table=TestTable [id=96744a1cdf364f2292e5ad0befa9d646]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:39.172626 24289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ed877410cc46fb8ab71a39bc9a39f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.172644 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ed877410cc46fb8ab71a39bc9a39f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.172644 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ed877410cc46fb8ab71a39bc9a39f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.177608 24372 tablet_bootstrap.cc:492] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a: Bootstrap starting.
I20240502 13:59:39.178670 24373 tablet_bootstrap.cc:492] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c: Bootstrap starting.
I20240502 13:59:39.179155 24374 tablet_bootstrap.cc:492] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b: Bootstrap starting.
I20240502 13:59:39.180240 24372 tablet_bootstrap.cc:654] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.180593 24373 tablet_bootstrap.cc:654] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.180912 24372 log.cc:824] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:39.180995 24374 tablet_bootstrap.cc:654] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.181342 24373 log.cc:824] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:39.181720 24374 log.cc:824] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:39.182636 24372 tablet_bootstrap.cc:492] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a: No bootstrap required, opened a new log
I20240502 13:59:39.182808 24372 ts_tablet_manager.cc:1367] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:39.182952 24373 tablet_bootstrap.cc:492] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c: No bootstrap required, opened a new log
I20240502 13:59:39.183130 24373 ts_tablet_manager.cc:1367] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:39.183858 24374 tablet_bootstrap.cc:492] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b: No bootstrap required, opened a new log
I20240502 13:59:39.184062 24374 ts_tablet_manager.cc:1367] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 13:59:39.186471 24373 raft_consensus.cc:348] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.186673 24373 raft_consensus.cc:374] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.186543 24372 raft_consensus.cc:348] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.186723 24373 raft_consensus.cc:729] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b6f6c967f5452cbbf470dc26af4d4c, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.186745 24372 raft_consensus.cc:374] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.186806 24372 raft_consensus.cc:729] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb2cb5bf7d34198bb2287df4eeaba3a, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.187162 24373 consensus_queue.cc:260] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.187309 24372 consensus_queue.cc:260] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.187930 24373 ts_tablet_manager.cc:1398] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:39.188072 24372 ts_tablet_manager.cc:1398] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240502 13:59:39.188264 23962 heartbeater.cc:503] Master 127.22.55.190:40859 was elected leader, sending a full tablet report...
I20240502 13:59:39.188369 24355 heartbeater.cc:503] Master 127.22.55.190:40859 was elected leader, sending a full tablet report...
I20240502 13:59:39.188834 24374 raft_consensus.cc:348] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.188978 24374 raft_consensus.cc:374] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.189038 24374 raft_consensus.cc:729] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f704db56294d6fae53be7563d4b07b, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.189411 24374 consensus_queue.cc:260] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.190148 24374 ts_tablet_manager.cc:1398] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20240502 13:59:39.190426 24093 heartbeater.cc:503] Master 127.22.55.190:40859 was elected leader, sending a full tablet report...
W20240502 13:59:39.268132 24094 tablet.cc:2369] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:39.341929 23963 tablet.cc:2369] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:39.382339 24356 tablet.cc:2369] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:39.426182 23831 heartbeater.cc:503] Master 127.22.55.190:40859 was elected leader, sending a full tablet report...
I20240502 13:59:39.449445 24380 raft_consensus.cc:482] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:39.449661 24380 raft_consensus.cc:504] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.450793 24380 leader_election.cc:288] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb2cb5bf7d34198bb2287df4eeaba3a (127.22.55.133:37699), 00b6f6c967f5452cbbf470dc26af4d4c (127.22.55.130:39157)
I20240502 13:59:39.454576 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ed877410cc46fb8ab71a39bc9a39f8" candidate_uuid: "44f704db56294d6fae53be7563d4b07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" is_pre_election: true
I20240502 13:59:39.454644 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ed877410cc46fb8ab71a39bc9a39f8" candidate_uuid: "44f704db56294d6fae53be7563d4b07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" is_pre_election: true
I20240502 13:59:39.454864 24309 raft_consensus.cc:2454] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f704db56294d6fae53be7563d4b07b in term 0.
I20240502 13:59:39.454967 23916 raft_consensus.cc:2454] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f704db56294d6fae53be7563d4b07b in term 0.
I20240502 13:59:39.455328 23981 leader_election.cc:302] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f704db56294d6fae53be7563d4b07b, 8eb2cb5bf7d34198bb2287df4eeaba3a; no voters:
I20240502 13:59:39.455554 24380 raft_consensus.cc:2781] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:39.455621 24380 raft_consensus.cc:482] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:39.455649 24380 raft_consensus.cc:3037] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.456483 24380 raft_consensus.cc:504] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.456804 24380 leader_election.cc:288] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [CANDIDATE]: Term 1 election: Requested vote from peers 8eb2cb5bf7d34198bb2287df4eeaba3a (127.22.55.133:37699), 00b6f6c967f5452cbbf470dc26af4d4c (127.22.55.130:39157)
I20240502 13:59:39.457192 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ed877410cc46fb8ab71a39bc9a39f8" candidate_uuid: "44f704db56294d6fae53be7563d4b07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a"
I20240502 13:59:39.457257 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ed877410cc46fb8ab71a39bc9a39f8" candidate_uuid: "44f704db56294d6fae53be7563d4b07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b6f6c967f5452cbbf470dc26af4d4c"
I20240502 13:59:39.457306 24309 raft_consensus.cc:3037] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.457345 23916 raft_consensus.cc:3037] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.458140 23916 raft_consensus.cc:2454] T c5ed877410cc46fb8ab71a39bc9a39f8 P 00b6f6c967f5452cbbf470dc26af4d4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f704db56294d6fae53be7563d4b07b in term 1.
I20240502 13:59:39.458146 24309 raft_consensus.cc:2454] T c5ed877410cc46fb8ab71a39bc9a39f8 P 8eb2cb5bf7d34198bb2287df4eeaba3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f704db56294d6fae53be7563d4b07b in term 1.
I20240502 13:59:39.458385 23981 leader_election.cc:302] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b6f6c967f5452cbbf470dc26af4d4c, 44f704db56294d6fae53be7563d4b07b; no voters:
I20240502 13:59:39.458531 24380 raft_consensus.cc:2781] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.458729 24380 raft_consensus.cc:686] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [term 1 LEADER]: Becoming Leader. State: Replica: 44f704db56294d6fae53be7563d4b07b, State: Running, Role: LEADER
I20240502 13:59:39.459054 24380 consensus_queue.cc:237] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } }
I20240502 13:59:39.461620 23642 catalog_manager.cc:5484] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f704db56294d6fae53be7563d4b07b (127.22.55.131). New cstate: current_term: 1 leader_uuid: "44f704db56294d6fae53be7563d4b07b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 } } peers { permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 } } peers { permanent_uuid: "44f704db56294d6fae53be7563d4b07b" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38469 } } }
I20240502 13:59:39.525290 22750 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 13:59:39.526517 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c3538f5fb7584c91b2a8d0ef82163780 to finish bootstrapping
I20240502 13:59:39.531980 22750 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 00b6f6c967f5452cbbf470dc26af4d4c to finish bootstrapping
I20240502 13:59:39.536348 22750 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 44f704db56294d6fae53be7563d4b07b to finish bootstrapping
I20240502 13:59:39.540627 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3ec7c1e7b3ed439bb678fe4df426f427 to finish bootstrapping
I20240502 13:59:39.545394 22750 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8eb2cb5bf7d34198bb2287df4eeaba3a to finish bootstrapping
I20240502 13:59:39.549706 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 23703
I20240502 13:59:39.729441 23896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.729830 24158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.731093 24289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.731091 24027 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:39.801849 23896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.802176 24027 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:39.802840 24158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.805586 24289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.894227 24380 consensus_queue.cc:1035] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb2cb5bf7d34198bb2287df4eeaba3a" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 37699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:39.900198 24383 consensus_queue.cc:1035] T c5ed877410cc46fb8ab71a39bc9a39f8 P 44f704db56294d6fae53be7563d4b07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b6f6c967f5452cbbf470dc26af4d4c" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:39.901460 23896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.904548 24158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.905009 24027 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:39.907720 24289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:39.958366 24224 heartbeater.cc:503] Master 127.22.55.190:40859 was elected leader, sending a full tablet report...
I20240502 13:59:40.115736 23896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.116624 24158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.117008 24027 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:40.117590 24289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.189858 24158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.192049 23896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.198673 24289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.199883 24027 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:40.298228 23896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.299755 24158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.299852 24289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:40.300261 24027 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:40.330181 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 23834
I20240502 13:59:40.335695 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 23965
I20240502 13:59:40.341233 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 24096
I20240502 13:59:40.346750 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 24227
I20240502 13:59:40.352056 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 23612
2024-05-02T13:59:40Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (2355 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2355 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240502 13:59:40.374578 22750 test_util.cc:268] Using random seed: 61628974
I20240502 13:59:40.375341 22750 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:40.375388 22750 ts_itest-base.cc:113] --------------
I20240502 13:59:40.375401 22750 ts_itest-base.cc:114] 7 tablet servers
I20240502 13:59:40.375411 22750 ts_itest-base.cc:115] 1 replicas per TS
I20240502 13:59:40.375433 22750 ts_itest-base.cc:116] --------------
2024-05-02T13:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:40Z Disabled control of system clock
I20240502 13:59:40.382977 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:38111
--webserver_interface=127.22.55.190
--webserver_port=0
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.55.190:38111
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 13:59:40.484392 24580 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 13:59:40.484611 24580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.484637 24580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.484673 24580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.487538 24580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:40.487596 24580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:40.487617 24580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.487644 24580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:40.487660 24580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:40.491139 24580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.55.190:38111
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.55.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.491925 24580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.492954 24580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.500461 24585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.501220 24586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.501910 24588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.502629 24580 server_base.cc:1034] running on GCE node
I20240502 13:59:40.503144 24580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.503994 24580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.505149 24580 hybrid_clock.cc:648] HybridClock initialized: now 1714658380505130 us; error 41 us; skew 500 ppm
I20240502 13:59:40.506923 24580 webserver.cc:460] Webserver started at http://127.22.55.190:39097/ using document root <none> and password file <none>
I20240502 13:59:40.507541 24580 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.507604 24580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.507781 24580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.509348 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "84497bd32f594599b786a42799ec8d7a"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.509805 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "84497bd32f594599b786a42799ec8d7a"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.513025 24580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 13:59:40.515250 24594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.516192 24580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:40.516304 24580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
uuid: "84497bd32f594599b786a42799ec8d7a"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.516402 24580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.551290 24580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.551904 24580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.552062 24580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.558188 24580 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.190:38111
I20240502 13:59:40.558197 24646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.190:38111 every 8 connection(s)
I20240502 13:59:40.559167 24580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 13:59:40.561578 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:40.566411 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a: Bootstrap starting.
I20240502 13:59:40.568208 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 24580
I20240502 13:59:40.568316 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 13:59:40.568600 24647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:40.569420 24647 log.cc:824] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:40.571329 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a: No bootstrap required, opened a new log
I20240502 13:59:40.574402 24647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84497bd32f594599b786a42799ec8d7a" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 38111 } }
I20240502 13:59:40.574611 24647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:40.574666 24647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84497bd32f594599b786a42799ec8d7a, State: Initialized, Role: FOLLOWER
I20240502 13:59:40.575137 24647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84497bd32f594599b786a42799ec8d7a" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 38111 } }
I20240502 13:59:40.575289 24647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:40.575364 24647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:40.575491 24647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:40.576452 24647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84497bd32f594599b786a42799ec8d7a" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 38111 } }
I20240502 13:59:40.576771 24647 leader_election.cc:302] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84497bd32f594599b786a42799ec8d7a; no voters:
I20240502 13:59:40.577028 24647 leader_election.cc:288] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:40.577426 24652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:40.577600 24652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [term 1 LEADER]: Becoming Leader. State: Replica: 84497bd32f594599b786a42799ec8d7a, State: Running, Role: LEADER
I20240502 13:59:40.577868 24652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84497bd32f594599b786a42799ec8d7a" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 38111 } }
I20240502 13:59:40.578199 24647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:40.579025 24654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84497bd32f594599b786a42799ec8d7a. Latest consensus state: current_term: 1 leader_uuid: "84497bd32f594599b786a42799ec8d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84497bd32f594599b786a42799ec8d7a" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 38111 } } }
I20240502 13:59:40.579152 24654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:40.579322 24653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84497bd32f594599b786a42799ec8d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84497bd32f594599b786a42799ec8d7a" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 38111 } } }
I20240502 13:59:40.579447 24653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:40.579530 24661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:40.582309 24661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:40.586871 24661 catalog_manager.cc:1347] Generated new cluster ID: 717e2947fbe344cc90752103c5b16706
I20240502 13:59:40.586938 24661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:40.600250 24661 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:40.601079 24661 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:40.608120 24661 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 84497bd32f594599b786a42799ec8d7a: Generated new TSK 0
I20240502 13:59:40.608718 24661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:40.612459 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.129:0
--local_ip_for_outbound_sockets=127.22.55.129
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 13:59:40.716197 24671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.716400 24671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.716444 24671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.719457 24671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:40.719518 24671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.719607 24671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.129
I20240502 13:59:40.723150 24671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.24671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.129
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.723986 24671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.725064 24671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.733809 24677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.734313 24679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.734998 24676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.736045 24671 server_base.cc:1034] running on GCE node
I20240502 13:59:40.736435 24671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.737035 24671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.738210 24671 hybrid_clock.cc:648] HybridClock initialized: now 1714658380738169 us; error 50 us; skew 500 ppm
I20240502 13:59:40.740234 24671 webserver.cc:460] Webserver started at http://127.22.55.129:41501/ using document root <none> and password file <none>
I20240502 13:59:40.740878 24671 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.740958 24671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.741168 24671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.743042 24671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7b4b1969054146119af3a958ef85de55"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.744272 24671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7b4b1969054146119af3a958ef85de55"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.747467 24671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:40.749480 24685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.750403 24671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:40.750491 24671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7b4b1969054146119af3a958ef85de55"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.750561 24671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.764079 24671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.764621 24671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.764750 24671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.765199 24671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.765991 24671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.766036 24671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.766074 24671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.766094 24671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.774123 24671 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.129:36649
I20240502 13:59:40.774159 24798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.129:36649 every 8 connection(s)
I20240502 13:59:40.775066 24671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 13:59:40.778141 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 24671
I20240502 13:59:40.778259 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 13:59:40.780537 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.130:0
--local_ip_for_outbound_sockets=127.22.55.130
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:40.784366 24799 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:40.784570 24799 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.784971 24799 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:40.786520 24611 ts_manager.cc:194] Registered new tserver with Master: 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:40.788347 24611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.129:38701
W20240502 13:59:40.885353 24802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.885555 24802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.885596 24802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.888509 24802 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:40.888567 24802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.888629 24802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.130
I20240502 13:59:40.892019 24802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.24802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.130
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.892767 24802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.893769 24802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.902103 24807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.902827 24810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.903589 24808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.904345 24802 server_base.cc:1034] running on GCE node
I20240502 13:59:40.904664 24802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.905150 24802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.906291 24802 hybrid_clock.cc:648] HybridClock initialized: now 1714658380906266 us; error 37 us; skew 500 ppm
I20240502 13:59:40.908010 24802 webserver.cc:460] Webserver started at http://127.22.55.130:40599/ using document root <none> and password file <none>
I20240502 13:59:40.908521 24802 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.908576 24802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.908747 24802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.910315 24802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6e4fcab25c864479b96f38d291ebedaf"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.910748 24802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6e4fcab25c864479b96f38d291ebedaf"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.915165 24802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 13:59:40.917225 24816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.918135 24802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:40.918233 24802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6e4fcab25c864479b96f38d291ebedaf"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-sdf5"
I20240502 13:59:40.918308 24802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.944049 24802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.944685 24802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.944834 24802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.945360 24802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.946216 24802 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.946272 24802 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.946345 24802 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.946381 24802 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.954285 24802 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.130:38051
I20240502 13:59:40.954324 24929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.130:38051 every 8 connection(s)
I20240502 13:59:40.955222 24802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 13:59:40.956627 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 24802
I20240502 13:59:40.956766 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 13:59:40.958951 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.131:0
--local_ip_for_outbound_sockets=127.22.55.131
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:40.964242 24930 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:40.964437 24930 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.964815 24930 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:40.965719 24611 ts_manager.cc:194] Registered new tserver with Master: 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:40.966266 24611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.130:37913
W20240502 13:59:41.061519 24933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:41.061707 24933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:41.061746 24933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:41.064611 24933 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:41.064668 24933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:41.064746 24933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.131
I20240502 13:59:41.068230 24933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.24933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.131
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:41.069033 24933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:41.070108 24933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:41.078423 24941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.079257 24938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.079917 24939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:41.080595 24933 server_base.cc:1034] running on GCE node
I20240502 13:59:41.080909 24933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:41.081414 24933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:41.082557 24933 hybrid_clock.cc:648] HybridClock initialized: now 1714658381082551 us; error 26 us; skew 500 ppm
I20240502 13:59:41.084281 24933 webserver.cc:460] Webserver started at http://127.22.55.131:38119/ using document root <none> and password file <none>
I20240502 13:59:41.084789 24933 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:41.084846 24933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:41.084997 24933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:41.086536 24933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "51d1006508074f10acd579729457b0c5"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.087024 24933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "51d1006508074f10acd579729457b0c5"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.090097 24933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:41.092085 24947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.093015 24933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:41.093125 24933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "51d1006508074f10acd579729457b0c5"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.093205 24933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:41.129137 24933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:41.129959 24933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:41.130163 24933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:41.130841 24933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:41.132135 24933 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:41.132205 24933 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.132267 24933 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:41.132308 24933 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.143417 24933 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.131:42509
I20240502 13:59:41.143445 25061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.131:42509 every 8 connection(s)
I20240502 13:59:41.144445 24933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:41.145278 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 24933
I20240502 13:59:41.145390 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 13:59:41.147624 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.132:0
--local_ip_for_outbound_sockets=127.22.55.132
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:41.155663 25062 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:41.155913 25062 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:41.156486 25062 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:41.157667 24611 ts_manager.cc:194] Registered new tserver with Master: 51d1006508074f10acd579729457b0c5 (127.22.55.131:42509)
I20240502 13:59:41.158449 24611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.131:46069
W20240502 13:59:41.255306 25065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:41.255493 25065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:41.255535 25065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:41.258483 25065 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:41.258536 25065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:41.258607 25065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.132
I20240502 13:59:41.262239 25065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.25065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.132
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:41.263062 25065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:41.264114 25065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:41.272446 25071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.273339 25073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.274015 25070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:41.274945 25065 server_base.cc:1034] running on GCE node
I20240502 13:59:41.275277 25065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:41.275836 25065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:41.276980 25065 hybrid_clock.cc:648] HybridClock initialized: now 1714658381276963 us; error 38 us; skew 500 ppm
I20240502 13:59:41.278717 25065 webserver.cc:460] Webserver started at http://127.22.55.132:41885/ using document root <none> and password file <none>
I20240502 13:59:41.279302 25065 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:41.279359 25065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:41.279541 25065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:41.281096 25065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8aa9c677a2c84c5082c708ae767eb3a8"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.281546 25065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8aa9c677a2c84c5082c708ae767eb3a8"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.285008 25065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:41.287019 25079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.287936 25065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:41.288026 25065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8aa9c677a2c84c5082c708ae767eb3a8"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.288118 25065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:41.320273 25065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:41.320839 25065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:41.320966 25065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:41.321429 25065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:41.322283 25065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:41.322329 25065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.322369 25065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:41.322391 25065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.330122 25065 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.132:39445
I20240502 13:59:41.330148 25192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.132:39445 every 8 connection(s)
I20240502 13:59:41.331082 25065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:41.334609 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 25065
I20240502 13:59:41.334719 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 13:59:41.336771 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.133:0
--local_ip_for_outbound_sockets=127.22.55.133
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:41.339912 25193 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:41.340126 25193 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:41.340529 25193 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:41.341503 24611 ts_manager.cc:194] Registered new tserver with Master: 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132:39445)
I20240502 13:59:41.342279 24611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.132:55311
W20240502 13:59:41.442499 25196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:41.442687 25196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:41.442727 25196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:41.445696 25196 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:41.445788 25196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:41.445921 25196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.133
I20240502 13:59:41.449426 25196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.25196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.133
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:41.450244 25196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:41.451337 25196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:41.459507 25201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.460161 25202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.461081 25204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:41.461867 25196 server_base.cc:1034] running on GCE node
I20240502 13:59:41.462205 25196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:41.462700 25196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:41.463858 25196 hybrid_clock.cc:648] HybridClock initialized: now 1714658381463859 us; error 45 us; skew 500 ppm
I20240502 13:59:41.470935 25196 webserver.cc:460] Webserver started at http://127.22.55.133:44025/ using document root <none> and password file <none>
I20240502 13:59:41.471477 25196 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:41.471537 25196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:41.471704 25196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:41.473305 25196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b2a245b6dbf34e03a431a50a7b54ce92"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.473770 25196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b2a245b6dbf34e03a431a50a7b54ce92"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.477031 25196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:41.479178 25210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.480167 25196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:41.480273 25196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b2a245b6dbf34e03a431a50a7b54ce92"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.480350 25196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:41.507174 25196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:41.507809 25196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:41.507963 25196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:41.508491 25196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:41.509354 25196 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:41.509403 25196 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.509464 25196 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:41.509490 25196 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.517905 25196 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.133:38561
I20240502 13:59:41.517930 25323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.133:38561 every 8 connection(s)
I20240502 13:59:41.518772 25196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:41.522805 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 25196
I20240502 13:59:41.522929 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 13:59:41.524979 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.134:0
--local_ip_for_outbound_sockets=127.22.55.134
--webserver_interface=127.22.55.134
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:41.527737 25324 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:41.527973 25324 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:41.528537 25324 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:41.529606 24611 ts_manager.cc:194] Registered new tserver with Master: b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133:38561)
I20240502 13:59:41.530455 24611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.133:54311
W20240502 13:59:41.634546 25327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:41.634734 25327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:41.634773 25327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:41.637748 25327 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:41.637805 25327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:41.637868 25327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.134
I20240502 13:59:41.641256 25327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.55.134
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.25327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.134
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:41.642030 25327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:41.643071 25327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:41.651204 25332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.652129 25335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.652803 25333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:41.653777 25327 server_base.cc:1034] running on GCE node
I20240502 13:59:41.654129 25327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:41.654670 25327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:41.655818 25327 hybrid_clock.cc:648] HybridClock initialized: now 1714658381655810 us; error 44 us; skew 500 ppm
I20240502 13:59:41.657486 25327 webserver.cc:460] Webserver started at http://127.22.55.134:41255/ using document root <none> and password file <none>
I20240502 13:59:41.657994 25327 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:41.658051 25327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:41.658218 25327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:41.659850 25327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c970e31aa29e4944ab5fb3024984aefd"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.660308 25327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c970e31aa29e4944ab5fb3024984aefd"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.663437 25327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:41.665500 25341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.666419 25327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:41.666524 25327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c970e31aa29e4944ab5fb3024984aefd"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.666600 25327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:41.678653 25327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:41.679258 25327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:41.679409 25327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:41.679872 25327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:41.680711 25327 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:41.680755 25327 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.680809 25327 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:41.680840 25327 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.688884 25327 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.134:41009
I20240502 13:59:41.688915 25454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.134:41009 every 8 connection(s)
I20240502 13:59:41.689699 25327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 13:59:41.690877 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 25327
I20240502 13:59:41.690987 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 13:59:41.693295 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.135:0
--local_ip_for_outbound_sockets=127.22.55.135
--webserver_interface=127.22.55.135
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:41.698900 25455 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:41.699088 25455 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:41.699470 25455 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:41.700325 24611 ts_manager.cc:194] Registered new tserver with Master: c970e31aa29e4944ab5fb3024984aefd (127.22.55.134:41009)
I20240502 13:59:41.700838 24611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.134:52301
I20240502 13:59:41.790505 24799 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
W20240502 13:59:41.795948 25458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:41.796149 25458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:41.796216 25458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:41.799137 25458 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:41.799201 25458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:41.799280 25458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.135
I20240502 13:59:41.802592 25458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.55.135
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.25458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.135
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:41.803334 25458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:41.804322 25458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:41.812376 25466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.812999 25464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:41.813920 25463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:41.814702 25458 server_base.cc:1034] running on GCE node
I20240502 13:59:41.815019 25458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:41.815491 25458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:41.816623 25458 hybrid_clock.cc:648] HybridClock initialized: now 1714658381816606 us; error 30 us; skew 500 ppm
I20240502 13:59:41.818250 25458 webserver.cc:460] Webserver started at http://127.22.55.135:34017/ using document root <none> and password file <none>
I20240502 13:59:41.818739 25458 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:41.818790 25458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:41.818987 25458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:41.820616 25458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a9e66f5a1b7f4202860a428d70203609"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.821053 25458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a9e66f5a1b7f4202860a428d70203609"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.824133 25458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:41.826048 25472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.826944 25458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:41.827044 25458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a9e66f5a1b7f4202860a428d70203609"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:41.827124 25458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:41.836381 25458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:41.836877 25458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:41.837020 25458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:41.837472 25458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:41.838225 25458 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:41.838268 25458 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.838310 25458 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:41.838339 25458 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:41.846036 25458 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.135:41681
I20240502 13:59:41.846064 25585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.135:41681 every 8 connection(s)
I20240502 13:59:41.846836 25458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 13:59:41.848734 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 25458
I20240502 13:59:41.848838 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 13:59:41.854766 25586 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:41.854990 25586 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:41.855365 25586 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:41.856195 24611 ts_manager.cc:194] Registered new tserver with Master: a9e66f5a1b7f4202860a428d70203609 (127.22.55.135:41681)
I20240502 13:59:41.856740 24611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.135:55753
I20240502 13:59:41.862834 22750 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 13:59:41.870993 24611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49964:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 13:59:41.885931 24864 tablet_service.cc:1467] Processing CreateTablet for tablet ade0bed961d448449904f531c57ad940 (DEFAULT_TABLE table=TestTable [id=e67d2bd1c0d84039b6c44b58e82916af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:41.886688 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade0bed961d448449904f531c57ad940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:41.891265 25600 tablet_bootstrap.cc:492] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Bootstrap starting.
I20240502 13:59:41.893306 25600 tablet_bootstrap.cc:654] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:41.893935 25600 log.cc:824] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:41.895331 25600 tablet_bootstrap.cc:492] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: No bootstrap required, opened a new log
I20240502 13:59:41.895466 25600 ts_tablet_manager.cc:1367] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 13:59:41.897851 25600 raft_consensus.cc:348] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:41.898007 25600 raft_consensus.cc:374] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:41.898043 25600 raft_consensus.cc:729] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Initialized, Role: FOLLOWER
I20240502 13:59:41.898352 25600 consensus_queue.cc:260] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:41.898463 25600 raft_consensus.cc:388] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:41.898504 25600 raft_consensus.cc:482] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:41.898574 25600 raft_consensus.cc:3037] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:41.899621 25600 raft_consensus.cc:504] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:41.899914 25600 leader_election.cc:302] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:41.900177 25600 leader_election.cc:288] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:41.900292 25602 raft_consensus.cc:2781] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:41.900485 25602 raft_consensus.cc:686] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Leader. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:41.900703 24930 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
I20240502 13:59:41.900723 25602 consensus_queue.cc:237] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:41.900913 25600 ts_tablet_manager.cc:1398] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240502 13:59:41.903533 24611 catalog_manager.cc:5484] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130). New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } }
I20240502 13:59:41.911265 22750 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 13:59:41.912351 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7b4b1969054146119af3a958ef85de55 to finish bootstrapping
I20240502 13:59:41.917632 22750 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6e4fcab25c864479b96f38d291ebedaf to finish bootstrapping
I20240502 13:59:41.921813 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 51d1006508074f10acd579729457b0c5 to finish bootstrapping
I20240502 13:59:41.926438 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8aa9c677a2c84c5082c708ae767eb3a8 to finish bootstrapping
I20240502 13:59:41.931293 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b2a245b6dbf34e03a431a50a7b54ce92 to finish bootstrapping
I20240502 13:59:41.935963 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c970e31aa29e4944ab5fb3024984aefd to finish bootstrapping
I20240502 13:59:41.940672 22750 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a9e66f5a1b7f4202860a428d70203609 to finish bootstrapping
I20240502 13:59:41.945132 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 24933
I20240502 13:59:41.950608 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 25065
I20240502 13:59:41.957027 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 25196
I20240502 13:59:41.962231 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 25327
I20240502 13:59:41.968465 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 25458
I20240502 13:59:45.724800 24611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49964:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:45.735654 24733 tablet_service.cc:1467] Processing CreateTablet for tablet d465839425704eef962eb345979233f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffa197925b7242d892b01e70a6bd7c89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.735654 24731 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe45162fdb94d61961fea1cb7cc6af8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffa197925b7242d892b01e70a6bd7c89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.735644 24732 tablet_service.cc:1467] Processing CreateTablet for tablet b81678740ddc4fce9da067dc3d8dff8d (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffa197925b7242d892b01e70a6bd7c89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.736699 24731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe45162fdb94d61961fea1cb7cc6af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.737341 24733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465839425704eef962eb345979233f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.738428 24732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81678740ddc4fce9da067dc3d8dff8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.746186 25620 tablet_bootstrap.cc:492] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:45.748173 25620 tablet_bootstrap.cc:654] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.748888 25620 log.cc:824] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:45.750300 25620 tablet_bootstrap.cc:492] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:45.750459 25620 ts_tablet_manager.cc:1367] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 13:59:45.752775 25620 raft_consensus.cc:348] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.752943 25620 raft_consensus.cc:374] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.753015 25620 raft_consensus.cc:729] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.753439 25620 consensus_queue.cc:260] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.753558 25620 raft_consensus.cc:388] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:45.753620 25620 raft_consensus.cc:482] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:45.753722 25620 raft_consensus.cc:3037] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.754580 25620 raft_consensus.cc:504] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.754909 25620 leader_election.cc:302] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:45.755163 25620 leader_election.cc:288] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:45.755239 25622 raft_consensus.cc:2781] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.755453 25622 raft_consensus.cc:686] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:45.755946 25620 ts_tablet_manager.cc:1398] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 13:59:45.756039 25622 consensus_queue.cc:237] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.756156 25620 tablet_bootstrap.cc:492] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:45.756843 25620 tablet_bootstrap.cc:654] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.757738 25620 tablet_bootstrap.cc:492] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:45.757813 25620 ts_tablet_manager.cc:1367] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:45.758355 25620 raft_consensus.cc:348] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.758432 25620 raft_consensus.cc:374] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.758467 25620 raft_consensus.cc:729] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.758605 25620 consensus_queue.cc:260] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.758685 25620 raft_consensus.cc:388] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:45.758617 24611 catalog_manager.cc:5484] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
I20240502 13:59:45.758733 25620 raft_consensus.cc:482] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:45.758780 25620 raft_consensus.cc:3037] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.759518 25620 raft_consensus.cc:504] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.759639 25620 leader_election.cc:302] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:45.759732 25620 leader_election.cc:288] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:45.759800 25623 raft_consensus.cc:2781] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.759898 25623 raft_consensus.cc:686] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:45.759972 25620 ts_tablet_manager.cc:1398] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.760012 25623 consensus_queue.cc:237] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.760102 25620 tablet_bootstrap.cc:492] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:45.760936 25620 tablet_bootstrap.cc:654] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.762055 24611 catalog_manager.cc:5484] T 4fe45162fdb94d61961fea1cb7cc6af8 P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
I20240502 13:59:45.762475 25620 tablet_bootstrap.cc:492] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:45.762555 25620 ts_tablet_manager.cc:1367] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:45.762985 25620 raft_consensus.cc:348] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.763057 25620 raft_consensus.cc:374] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.763078 25620 raft_consensus.cc:729] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.763209 25620 consensus_queue.cc:260] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.763303 25620 raft_consensus.cc:388] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:45.763342 25620 raft_consensus.cc:482] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:45.763387 25620 raft_consensus.cc:3037] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.764055 25620 raft_consensus.cc:504] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.764151 25620 leader_election.cc:302] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:45.764219 25620 leader_election.cc:288] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:45.764308 25622 raft_consensus.cc:2781] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.764394 25622 raft_consensus.cc:686] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:45.764456 25620 ts_tablet_manager.cc:1398] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:45.764568 25622 consensus_queue.cc:237] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:45.765586 24611 catalog_manager.cc:5484] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
W20240502 13:59:45.871771 25631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:45.872068 25631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:45.875216 25631 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 61.1015 ms
time per row: 0.0305507 ms
I20240502 13:59:45.976186 24611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49964:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:45.981642 24863 tablet_service.cc:1467] Processing CreateTablet for tablet cae053fe22fd451e88e8e01eb93b76c5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e40cc0c516264dca942c3242c2fdad90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.981644 24864 tablet_service.cc:1467] Processing CreateTablet for tablet 00c8e8a7c7a6493088d08577906bf21c (DEFAULT_TABLE table=rebalance_test_table_1 [id=e40cc0c516264dca942c3242c2fdad90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.981644 24862 tablet_service.cc:1467] Processing CreateTablet for tablet 27fa46c3e6214af2a67805b7dcfd9301 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e40cc0c516264dca942c3242c2fdad90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.981969 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae053fe22fd451e88e8e01eb93b76c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.982098 24862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fa46c3e6214af2a67805b7dcfd9301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.982266 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c8e8a7c7a6493088d08577906bf21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.984041 25663 tablet_bootstrap.cc:492] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: Bootstrap starting.
I20240502 13:59:45.984967 25663 tablet_bootstrap.cc:654] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.985855 25663 tablet_bootstrap.cc:492] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: No bootstrap required, opened a new log
I20240502 13:59:45.985924 25663 ts_tablet_manager.cc:1367] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:45.986438 25663 raft_consensus.cc:348] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.986542 25663 raft_consensus.cc:374] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.986575 25663 raft_consensus.cc:729] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.986701 25663 consensus_queue.cc:260] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.986794 25663 raft_consensus.cc:388] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:45.986835 25663 raft_consensus.cc:482] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:45.986934 25663 raft_consensus.cc:3037] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.987550 25663 raft_consensus.cc:504] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.987679 25663 leader_election.cc:302] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:45.987824 25663 leader_election.cc:288] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:45.987916 25665 raft_consensus.cc:2781] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.988119 25663 ts_tablet_manager.cc:1398] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 13:59:45.988143 25665 raft_consensus.cc:686] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Leader. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:45.988274 25663 tablet_bootstrap.cc:492] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf: Bootstrap starting.
I20240502 13:59:45.988251 25665 consensus_queue.cc:237] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.988943 25663 tablet_bootstrap.cc:654] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.989706 24611 catalog_manager.cc:5484] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130). New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } }
I20240502 13:59:45.990792 25663 tablet_bootstrap.cc:492] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf: No bootstrap required, opened a new log
I20240502 13:59:45.990916 25663 ts_tablet_manager.cc:1367] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 13:59:45.991328 25663 raft_consensus.cc:348] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.991407 25663 raft_consensus.cc:374] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.991441 25663 raft_consensus.cc:729] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.991585 25663 consensus_queue.cc:260] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.991647 25663 raft_consensus.cc:388] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:45.991670 25663 raft_consensus.cc:482] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:45.991719 25663 raft_consensus.cc:3037] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.992647 25663 raft_consensus.cc:504] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.992758 25663 leader_election.cc:302] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:45.992923 25667 raft_consensus.cc:2781] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.992988 25663 leader_election.cc:288] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:45.993055 25667 raft_consensus.cc:686] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Leader. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:45.993186 25667 consensus_queue.cc:237] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.993238 25663 ts_tablet_manager.cc:1398] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.993506 25663 tablet_bootstrap.cc:492] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: Bootstrap starting.
I20240502 13:59:45.994243 25663 tablet_bootstrap.cc:654] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.994382 24611 catalog_manager.cc:5484] T 00c8e8a7c7a6493088d08577906bf21c P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130). New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } }
I20240502 13:59:45.995286 25663 tablet_bootstrap.cc:492] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: No bootstrap required, opened a new log
I20240502 13:59:45.995360 25663 ts_tablet_manager.cc:1367] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 13:59:45.995728 25663 raft_consensus.cc:348] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.995815 25663 raft_consensus.cc:374] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.995888 25663 raft_consensus.cc:729] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.996059 25663 consensus_queue.cc:260] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.996150 25663 raft_consensus.cc:388] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:45.996198 25663 raft_consensus.cc:482] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:45.996251 25663 raft_consensus.cc:3037] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.997035 25663 raft_consensus.cc:504] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.997156 25663 leader_election.cc:302] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:45.997253 25663 leader_election.cc:288] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:45.997395 25666 raft_consensus.cc:2781] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.997478 25663 ts_tablet_manager.cc:1398] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 13:59:45.997484 25666 raft_consensus.cc:686] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Leader. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:45.997783 25666 consensus_queue.cc:237] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:45.998816 24611 catalog_manager.cc:5484] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130). New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } }
W20240502 13:59:46.110075 25674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:46.110303 25674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:46.113250 25674 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 62.4423 ms
time per row: 0.0312212 ms
I20240502 13:59:46.214892 24611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49964:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:46.220125 24733 tablet_service.cc:1467] Processing CreateTablet for tablet 94738674b9af470f922b593daec77442 (DEFAULT_TABLE table=rebalance_test_table_2 [id=135ed63340794251874287c06a0d8b85]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.220222 24731 tablet_service.cc:1467] Processing CreateTablet for tablet 0cad80f3ea304711b302663a1f420aba (DEFAULT_TABLE table=rebalance_test_table_2 [id=135ed63340794251874287c06a0d8b85]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.220222 24732 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca9e7a8cb7644f48a634b8ef86cad9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=135ed63340794251874287c06a0d8b85]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.220492 24733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94738674b9af470f922b593daec77442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.220664 24731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cad80f3ea304711b302663a1f420aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.220801 24732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca9e7a8cb7644f48a634b8ef86cad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.222471 25620 tablet_bootstrap.cc:492] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:46.223209 25620 tablet_bootstrap.cc:654] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.224295 25620 tablet_bootstrap.cc:492] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:46.224386 25620 ts_tablet_manager.cc:1367] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.225044 25620 raft_consensus.cc:348] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.225143 25620 raft_consensus.cc:374] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.225175 25620 raft_consensus.cc:729] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.225270 25620 consensus_queue.cc:260] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.225325 25620 raft_consensus.cc:388] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.225353 25620 raft_consensus.cc:482] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.225387 25620 raft_consensus.cc:3037] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.225945 25620 raft_consensus.cc:504] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.226050 25620 leader_election.cc:302] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:46.226130 25620 leader_election.cc:288] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.226189 25622 raft_consensus.cc:2781] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.226274 25622 raft_consensus.cc:686] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:46.226382 25620 ts_tablet_manager.cc:1398] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:46.226390 25622 consensus_queue.cc:237] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.226545 25620 tablet_bootstrap.cc:492] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:46.227460 25620 tablet_bootstrap.cc:654] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.228133 24611 catalog_manager.cc:5484] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
I20240502 13:59:46.228469 25620 tablet_bootstrap.cc:492] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:46.228541 25620 ts_tablet_manager.cc:1367] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.228866 25620 raft_consensus.cc:348] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.228938 25620 raft_consensus.cc:374] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.228971 25620 raft_consensus.cc:729] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.229086 25620 consensus_queue.cc:260] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.229166 25620 raft_consensus.cc:388] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.229197 25620 raft_consensus.cc:482] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.229234 25620 raft_consensus.cc:3037] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.229959 25620 raft_consensus.cc:504] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.230072 25620 leader_election.cc:302] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:46.230192 25620 leader_election.cc:288] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.230237 25622 raft_consensus.cc:2781] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.230319 25622 raft_consensus.cc:686] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:46.230409 25620 ts_tablet_manager.cc:1398] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:46.230425 25622 consensus_queue.cc:237] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.230577 25620 tablet_bootstrap.cc:492] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:46.231343 25620 tablet_bootstrap.cc:654] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.232270 25620 tablet_bootstrap.cc:492] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:46.232334 25620 ts_tablet_manager.cc:1367] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.232282 24607 catalog_manager.cc:5484] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
I20240502 13:59:46.232702 25620 raft_consensus.cc:348] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.232789 25620 raft_consensus.cc:374] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.232820 25620 raft_consensus.cc:729] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.232928 25620 consensus_queue.cc:260] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.233009 25620 raft_consensus.cc:388] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.233049 25620 raft_consensus.cc:482] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.233250 25620 raft_consensus.cc:3037] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.233994 25620 raft_consensus.cc:504] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.234122 25620 leader_election.cc:302] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:46.234215 25620 leader_election.cc:288] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.234280 25622 raft_consensus.cc:2781] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.234361 25622 raft_consensus.cc:686] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:46.234426 25620 ts_tablet_manager.cc:1398] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.234450 25622 consensus_queue.cc:237] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.235514 24607 catalog_manager.cc:5484] T 0cad80f3ea304711b302663a1f420aba P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
W20240502 13:59:46.345099 25713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:46.345327 25713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:46.348246 25713 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 54.9809 ms
time per row: 0.0274905 ms
I20240502 13:59:46.441915 24611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49964:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:46.446985 24862 tablet_service.cc:1467] Processing CreateTablet for tablet ff8c1d90d5d8449aac5e3e1e81d54523 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a9389b0cdd3841079087847cdd44fb43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.447038 24863 tablet_service.cc:1467] Processing CreateTablet for tablet 94e5deba02294d4e85fd4bad586a78d5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a9389b0cdd3841079087847cdd44fb43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.446985 24864 tablet_service.cc:1467] Processing CreateTablet for tablet 07dbaad8ae36426b9eff6ea62db21dc7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a9389b0cdd3841079087847cdd44fb43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.447432 24862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8c1d90d5d8449aac5e3e1e81d54523. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.447597 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07dbaad8ae36426b9eff6ea62db21dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.449005 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e5deba02294d4e85fd4bad586a78d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.449118 25663 tablet_bootstrap.cc:492] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: Bootstrap starting.
I20240502 13:59:46.449918 25663 tablet_bootstrap.cc:654] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.451020 25663 tablet_bootstrap.cc:492] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: No bootstrap required, opened a new log
I20240502 13:59:46.451088 25663 ts_tablet_manager.cc:1367] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 13:59:46.451417 25663 raft_consensus.cc:348] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.451498 25663 raft_consensus.cc:374] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.451519 25663 raft_consensus.cc:729] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.451622 25663 consensus_queue.cc:260] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.451692 25663 raft_consensus.cc:388] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.451721 25663 raft_consensus.cc:482] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.451754 25663 raft_consensus.cc:3037] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.452309 25663 raft_consensus.cc:504] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.452415 25663 leader_election.cc:302] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:46.452486 25663 leader_election.cc:288] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.452553 25666 raft_consensus.cc:2781] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.452642 25666 raft_consensus.cc:686] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Leader. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:46.452745 25666 consensus_queue.cc:237] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.452978 25663 ts_tablet_manager.cc:1398] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:46.453462 25663 tablet_bootstrap.cc:492] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf: Bootstrap starting.
I20240502 13:59:46.453915 24611 catalog_manager.cc:5484] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130). New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } }
I20240502 13:59:46.454201 25663 tablet_bootstrap.cc:654] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.455175 25663 tablet_bootstrap.cc:492] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf: No bootstrap required, opened a new log
I20240502 13:59:46.455256 25663 ts_tablet_manager.cc:1367] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.455593 25663 raft_consensus.cc:348] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.455664 25663 raft_consensus.cc:374] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.455685 25663 raft_consensus.cc:729] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.455785 25663 consensus_queue.cc:260] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.455848 25663 raft_consensus.cc:388] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.455884 25663 raft_consensus.cc:482] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.455940 25663 raft_consensus.cc:3037] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.456965 25663 raft_consensus.cc:504] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.457080 25663 leader_election.cc:302] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:46.457195 25663 leader_election.cc:288] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.457252 25667 raft_consensus.cc:2781] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.457372 25667 raft_consensus.cc:686] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Leader. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:46.457433 25663 ts_tablet_manager.cc:1398] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:46.457506 25667 consensus_queue.cc:237] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.457595 25663 tablet_bootstrap.cc:492] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: Bootstrap starting.
I20240502 13:59:46.458374 25663 tablet_bootstrap.cc:654] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.459468 25663 tablet_bootstrap.cc:492] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: No bootstrap required, opened a new log
I20240502 13:59:46.459425 24611 catalog_manager.cc:5484] T 07dbaad8ae36426b9eff6ea62db21dc7 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130). New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } }
I20240502 13:59:46.459546 25663 ts_tablet_manager.cc:1367] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.459945 25663 raft_consensus.cc:348] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.460038 25663 raft_consensus.cc:374] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.460078 25663 raft_consensus.cc:729] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.460197 25663 consensus_queue.cc:260] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.460283 25663 raft_consensus.cc:388] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.460319 25663 raft_consensus.cc:482] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.460362 25663 raft_consensus.cc:3037] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.461112 25663 raft_consensus.cc:504] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.461248 25663 leader_election.cc:302] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:46.461349 25663 leader_election.cc:288] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.461400 25666 raft_consensus.cc:2781] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.461473 25666 raft_consensus.cc:686] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Leader. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:46.461580 25663 ts_tablet_manager.cc:1398] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 13:59:46.461580 25666 consensus_queue.cc:237] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:46.462620 24611 catalog_manager.cc:5484] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130). New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } }
W20240502 13:59:46.573812 25743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:46.574040 25743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:46.577136 25743 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 64.8719 ms
time per row: 0.0324359 ms
I20240502 13:59:46.691076 24603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49964:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:46.696476 24732 tablet_service.cc:1467] Processing CreateTablet for tablet 75834b9995394a2d95a56acc95eea043 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a4b94a9885349e8bb805979c6d23fb0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.696563 24731 tablet_service.cc:1467] Processing CreateTablet for tablet f943154c90654cd39f602a0fc2b90ceb (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a4b94a9885349e8bb805979c6d23fb0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.696496 24733 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae2eb2957c749b4a34b07944a5c3592 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2a4b94a9885349e8bb805979c6d23fb0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.696816 24732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75834b9995394a2d95a56acc95eea043. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.696979 24733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae2eb2957c749b4a34b07944a5c3592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.697188 24731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f943154c90654cd39f602a0fc2b90ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.698719 25620 tablet_bootstrap.cc:492] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:46.699615 25620 tablet_bootstrap.cc:654] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.700465 25620 tablet_bootstrap.cc:492] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:46.700536 25620 ts_tablet_manager.cc:1367] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.700906 25620 raft_consensus.cc:348] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.700989 25620 raft_consensus.cc:374] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.701025 25620 raft_consensus.cc:729] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.701153 25620 consensus_queue.cc:260] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.701217 25620 raft_consensus.cc:388] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.701252 25620 raft_consensus.cc:482] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.701299 25620 raft_consensus.cc:3037] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.783299 25620 raft_consensus.cc:504] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.783491 25620 leader_election.cc:302] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:46.783617 25620 leader_election.cc:288] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.783680 25622 raft_consensus.cc:2781] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.783838 25622 raft_consensus.cc:686] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:46.783974 25622 consensus_queue.cc:237] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.784019 25620 ts_tablet_manager.cc:1398] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.083s user 0.002s sys 0.000s
I20240502 13:59:46.784194 25620 tablet_bootstrap.cc:492] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:46.785187 25620 tablet_bootstrap.cc:654] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.785576 24603 catalog_manager.cc:5484] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
I20240502 13:59:46.786840 25620 tablet_bootstrap.cc:492] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:46.786983 25620 ts_tablet_manager.cc:1367] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:46.787415 25620 raft_consensus.cc:348] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.787556 25620 raft_consensus.cc:374] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.787628 25620 raft_consensus.cc:729] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.787782 25620 consensus_queue.cc:260] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.787894 25620 raft_consensus.cc:388] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.787966 25620 raft_consensus.cc:482] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.788017 25620 raft_consensus.cc:3037] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.880335 25620 raft_consensus.cc:504] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.880553 25620 leader_election.cc:302] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:46.880679 25620 leader_election.cc:288] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.880791 25768 raft_consensus.cc:2781] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.880918 25768 raft_consensus.cc:686] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:46.881098 25620 ts_tablet_manager.cc:1398] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.094s user 0.002s sys 0.000s
I20240502 13:59:46.881075 25768 consensus_queue.cc:237] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.881253 25620 tablet_bootstrap.cc:492] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: Bootstrap starting.
I20240502 13:59:46.882527 25620 tablet_bootstrap.cc:654] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.883395 24603 catalog_manager.cc:5484] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
I20240502 13:59:46.883682 25620 tablet_bootstrap.cc:492] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: No bootstrap required, opened a new log
I20240502 13:59:46.883765 25620 ts_tablet_manager.cc:1367] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:46.884191 25620 raft_consensus.cc:348] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.884286 25620 raft_consensus.cc:374] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.884325 25620 raft_consensus.cc:729] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.884449 25620 consensus_queue.cc:260] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.884538 25620 raft_consensus.cc:388] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:46.884574 25620 raft_consensus.cc:482] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:46.884616 25620 raft_consensus.cc:3037] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.982218 25620 raft_consensus.cc:504] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.982427 25620 leader_election.cc:302] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:46.982553 25620 leader_election.cc:288] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:46.982630 25768 raft_consensus.cc:2781] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.982771 25768 raft_consensus.cc:686] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:46.982837 25620 ts_tablet_manager.cc:1398] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: Time spent starting tablet: real 0.099s user 0.002s sys 0.000s
I20240502 13:59:46.982954 25768 consensus_queue.cc:237] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:46.984623 24603 catalog_manager.cc:5484] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4b1969054146119af3a958ef85de55 (127.22.55.129). New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } }
W20240502 13:59:47.163756 25774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.164005 25774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.167428 25774 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 96.0547 ms
time per row: 0.0480274 ms
I20240502 13:59:47.306624 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.131:42509
--local_ip_for_outbound_sockets=127.22.55.131
--tserver_master_addrs=127.22.55.190:38111
--webserver_port=38119
--webserver_interface=127.22.55.131
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 13:59:47.418035 25805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.418251 25805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:47.418294 25805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.421558 25805 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:47.421634 25805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:47.421728 25805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.131
I20240502 13:59:47.425446 25805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.131:42509
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.55.131
--webserver_port=38119
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.25805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.131
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:47.426328 25805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:47.427567 25805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:47.436319 25810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:47.437264 25813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:47.438109 25805 server_base.cc:1034] running on GCE node
W20240502 13:59:47.438948 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:47.439298 25805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:47.439810 25805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:47.440948 25805 hybrid_clock.cc:648] HybridClock initialized: now 1714658387440931 us; error 38 us; skew 500 ppm
I20240502 13:59:47.442721 25805 webserver.cc:460] Webserver started at http://127.22.55.131:38119/ using document root <none> and password file <none>
I20240502 13:59:47.443277 25805 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:47.443351 25805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:47.446516 25805 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:47.448354 25819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.449280 25805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:47.449394 25805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "51d1006508074f10acd579729457b0c5"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:47.449787 25805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:47.471504 25805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:47.472095 25805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:47.472255 25805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:47.472754 25805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:47.473598 25805 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:47.473647 25805 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.473707 25805 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:47.473734 25805 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.482311 25805 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.131:42509
I20240502 13:59:47.482375 25932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.131:42509 every 8 connection(s)
I20240502 13:59:47.483376 25805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:47.491892 25933 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:47.492117 25933 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:47.492569 25933 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:47.492954 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 25805
I20240502 13:59:47.493398 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.132:39445
--local_ip_for_outbound_sockets=127.22.55.132
--tserver_master_addrs=127.22.55.190:38111
--webserver_port=41885
--webserver_interface=127.22.55.132
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:47.493670 24603 ts_manager.cc:194] Re-registered known tserver with Master: 51d1006508074f10acd579729457b0c5 (127.22.55.131:42509)
I20240502 13:59:47.494248 24603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.131:50753
W20240502 13:59:47.603238 25936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.603459 25936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:47.603502 25936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.606479 25936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:47.606544 25936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:47.606609 25936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.132
I20240502 13:59:47.610971 25936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.132:39445
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.55.132
--webserver_port=41885
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.25936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.132
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:47.611879 25936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:47.612978 25936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:47.621785 25942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:47.622633 25944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:47.623378 25941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:47.624239 25936 server_base.cc:1034] running on GCE node
I20240502 13:59:47.624599 25936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:47.625114 25936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:47.626272 25936 hybrid_clock.cc:648] HybridClock initialized: now 1714658387626253 us; error 37 us; skew 500 ppm
I20240502 13:59:47.628055 25936 webserver.cc:460] Webserver started at http://127.22.55.132:41885/ using document root <none> and password file <none>
I20240502 13:59:47.628589 25936 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:47.628664 25936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:47.631909 25936 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20240502 13:59:47.633673 25950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.634568 25936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:47.634668 25936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8aa9c677a2c84c5082c708ae767eb3a8"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:47.635115 25936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:47.659088 25936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:47.659690 25936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:47.659848 25936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:47.660358 25936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:47.661193 25936 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:47.661242 25936 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.661299 25936 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:47.661329 25936 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.669147 25936 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.132:39445
I20240502 13:59:47.669188 26063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.132:39445 every 8 connection(s)
I20240502 13:59:47.670106 25936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:47.678699 26064 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:47.678948 26064 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:47.679417 26064 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:47.680387 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 25936
I20240502 13:59:47.680545 24603 ts_manager.cc:194] Re-registered known tserver with Master: 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132:39445)
I20240502 13:59:47.680809 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.133:38561
--local_ip_for_outbound_sockets=127.22.55.133
--tserver_master_addrs=127.22.55.190:38111
--webserver_port=44025
--webserver_interface=127.22.55.133
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:47.681294 24603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.132:52217
W20240502 13:59:47.786936 26067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.787137 26067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:47.787210 26067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.790131 26067 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:47.790195 26067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:47.790284 26067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.133
I20240502 13:59:47.793711 26067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.133:38561
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.55.133
--webserver_port=44025
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.26067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.133
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:47.794509 26067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:47.795646 26067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:47.803818 26075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:47.804661 26072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:47.805393 26073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:47.806233 26067 server_base.cc:1034] running on GCE node
I20240502 13:59:47.806535 26067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:47.807021 26067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:47.808162 26067 hybrid_clock.cc:648] HybridClock initialized: now 1714658387808147 us; error 27 us; skew 500 ppm
I20240502 13:59:47.809763 26067 webserver.cc:460] Webserver started at http://127.22.55.133:44025/ using document root <none> and password file <none>
I20240502 13:59:47.810297 26067 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:47.810362 26067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:47.813493 26067 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:47.815238 26081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.816195 26067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:47.816287 26067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b2a245b6dbf34e03a431a50a7b54ce92"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:47.816675 26067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:47.839198 26067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:47.839808 26067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:47.839938 26067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:47.840400 26067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:47.841215 26067 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:47.841261 26067 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.841300 26067 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:47.841320 26067 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.849141 26067 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.133:38561
I20240502 13:59:47.849193 26194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.133:38561 every 8 connection(s)
I20240502 13:59:47.850126 26067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:47.856303 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 26067
I20240502 13:59:47.856662 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.134:41009
--local_ip_for_outbound_sockets=127.22.55.134
--tserver_master_addrs=127.22.55.190:38111
--webserver_port=41255
--webserver_interface=127.22.55.134
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:47.858444 26195 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:47.858671 26195 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:47.859184 26195 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:47.860136 24603 ts_manager.cc:194] Re-registered known tserver with Master: b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133:38561)
I20240502 13:59:47.860661 24603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.133:46969
W20240502 13:59:47.964635 26198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.964836 26198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:47.964887 26198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.967887 26198 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:47.967952 26198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:47.968051 26198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.134
I20240502 13:59:47.971679 26198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.134:41009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.55.134
--webserver_port=41255
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.26198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.134
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:47.972481 26198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:47.973531 26198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:47.982141 26206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:47.982829 26204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:47.983880 26203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:47.984654 26198 server_base.cc:1034] running on GCE node
I20240502 13:59:47.985003 26198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:47.985527 26198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:47.986670 26198 hybrid_clock.cc:648] HybridClock initialized: now 1714658387986657 us; error 27 us; skew 500 ppm
I20240502 13:59:47.988487 26198 webserver.cc:460] Webserver started at http://127.22.55.134:41255/ using document root <none> and password file <none>
I20240502 13:59:47.989070 26198 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:47.989162 26198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:47.992545 26198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:47.994240 26212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:47.995245 26198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:47.995347 26198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c970e31aa29e4944ab5fb3024984aefd"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:47.995751 26198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:48.022223 26198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:48.022943 26198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:48.023108 26198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:48.023649 26198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:48.024510 26198 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:48.024556 26198 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:48.024601 26198 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:48.024632 26198 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:48.032891 26198 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.134:41009
I20240502 13:59:48.032914 26325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.134:41009 every 8 connection(s)
I20240502 13:59:48.033907 26198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 13:59:48.042469 26326 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:48.042713 26326 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:48.043130 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 26198
I20240502 13:59:48.043269 26326 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:48.043577 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.135:41681
--local_ip_for_outbound_sockets=127.22.55.135
--tserver_master_addrs=127.22.55.190:38111
--webserver_port=34017
--webserver_interface=127.22.55.135
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 13:59:48.044481 24603 ts_manager.cc:194] Re-registered known tserver with Master: c970e31aa29e4944ab5fb3024984aefd (127.22.55.134:41009)
I20240502 13:59:48.045230 24603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.134:58833
W20240502 13:59:48.150106 26329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:48.150300 26329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:48.150339 26329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:48.153391 26329 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 13:59:48.153450 26329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:48.153543 26329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.135
I20240502 13:59:48.157126 26329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.135:41681
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.55.135
--webserver_port=34017
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.26329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.135
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:48.157940 26329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:48.159035 26329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:48.167615 26334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:48.168545 26335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:48.169262 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:48.170166 26329 server_base.cc:1034] running on GCE node
I20240502 13:59:48.170500 26329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:48.171022 26329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:48.172180 26329 hybrid_clock.cc:648] HybridClock initialized: now 1714658388172169 us; error 24 us; skew 500 ppm
I20240502 13:59:48.173812 26329 webserver.cc:460] Webserver started at http://127.22.55.135:34017/ using document root <none> and password file <none>
I20240502 13:59:48.174338 26329 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:48.174419 26329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:48.177728 26329 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:48.179697 26343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:48.180685 26329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:48.180799 26329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a9e66f5a1b7f4202860a428d70203609"
format_stamp: "Formatted at 2024-05-02 13:59:41 on dist-test-slave-sdf5"
I20240502 13:59:48.181182 26329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:48.210758 26329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:48.211393 26329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:48.211547 26329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:48.212054 26329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:48.212932 26329 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:48.212983 26329 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:48.213048 26329 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:48.213076 26329 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:48.221519 26329 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.135:41681
I20240502 13:59:48.221561 26456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.135:41681 every 8 connection(s)
I20240502 13:59:48.222566 26329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 13:59:48.230691 26457 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 13:59:48.230914 26457 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:48.230901 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 26329
I20240502 13:59:48.231320 26457 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 13:59:48.232201 24603 ts_manager.cc:194] Re-registered known tserver with Master: a9e66f5a1b7f4202860a428d70203609 (127.22.55.135:41681)
I20240502 13:59:48.232715 24603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.135:37031
I20240502 13:59:48.415750 24732 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:48.421805 26391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.422230 25867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.424577 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.430119 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.430241 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.451289 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:48.496428 25933 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
I20240502 13:59:48.544080 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:48.545433 24732 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:48.549765 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.552712 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.553462 26391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.555279 25867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.564219 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:48.614356 24753 consensus_queue.cc:237] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
W20240502 13:59:48.619645 24686 consensus_peers.cc:485] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 -> Peer b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133:38561): Couldn't send request to peer b2a245b6dbf34e03a431a50a7b54ce92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d465839425704eef962eb345979233f6. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:48.683439 26064 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
I20240502 13:59:48.862927 26195 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
I20240502 13:59:49.047469 26326 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
I20240502 13:59:49.224138 26560 ts_tablet_manager.cc:902] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: Initiating tablet copy from peer 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:49.224850 26560 tablet_copy_client.cc:308] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:36649
I20240502 13:59:49.228973 24773 tablet_copy_service.cc:140] P 7b4b1969054146119af3a958ef85de55: Received BeginTabletCopySession request for tablet d465839425704eef962eb345979233f6 from peer b2a245b6dbf34e03a431a50a7b54ce92 ({username='slave'} at 127.22.55.133:57711)
I20240502 13:59:49.229102 24773 tablet_copy_service.cc:161] P 7b4b1969054146119af3a958ef85de55: Beginning new tablet copy session on tablet d465839425704eef962eb345979233f6 from peer b2a245b6dbf34e03a431a50a7b54ce92 at {username='slave'} at 127.22.55.133:57711: session id = b2a245b6dbf34e03a431a50a7b54ce92-d465839425704eef962eb345979233f6
I20240502 13:59:49.230319 24773 tablet_copy_source_session.cc:205] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:49.233387 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d465839425704eef962eb345979233f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.234352 26457 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
I20240502 13:59:49.238557 26560 tablet_copy_client.cc:783] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:49.238726 26560 tablet_copy_client.cc:647] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:49.240330 26560 tablet_copy_client.cc:523] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:49.246680 26560 tablet_bootstrap.cc:492] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: Bootstrap starting.
I20240502 13:59:49.252775 26560 log.cc:824] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:49.272395 26560 tablet_bootstrap.cc:492] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:49.272789 26560 tablet_bootstrap.cc:492] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: Bootstrap complete.
I20240502 13:59:49.273069 26560 ts_tablet_manager.cc:1367] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240502 13:59:49.275116 26560 raft_consensus.cc:348] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:49.275656 26560 raft_consensus.cc:729] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a245b6dbf34e03a431a50a7b54ce92, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.275931 26560 consensus_queue.cc:260] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:49.276762 26560 ts_tablet_manager.cc:1398] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 13:59:49.277683 24773 tablet_copy_service.cc:342] P 7b4b1969054146119af3a958ef85de55: Request end of tablet copy session b2a245b6dbf34e03a431a50a7b54ce92-d465839425704eef962eb345979233f6 received from {username='slave'} at 127.22.55.133:57711
I20240502 13:59:49.277770 24773 tablet_copy_service.cc:433] P 7b4b1969054146119af3a958ef85de55: ending tablet copy session b2a245b6dbf34e03a431a50a7b54ce92-d465839425704eef962eb345979233f6 on tablet d465839425704eef962eb345979233f6 with peer b2a245b6dbf34e03a431a50a7b54ce92
I20240502 13:59:49.846140 26149 raft_consensus.cc:1203] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:49.847532 26571 raft_consensus.cc:2932] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:49.848091 26149 raft_consensus.cc:2932] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:49.851914 26571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59732 (request call id 0) took 1238 ms. Trace:
I20240502 13:59:49.852041 26571 rpcz_store.cc:276] 0502 13:59:48.613735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 13:59:48.613798 (+ 63us) service_pool.cc:225] Handling call
0502 13:59:48.615235 (+ 1437us) log.cc:842] Serialized 260 byte log entry
0502 13:59:49.851894 (+1236659us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":51,"raft.queue_time_us":114,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":265,"spinlock_wait_cycles":21248,"thread_start_us":117,"threads_started":2,"wal-append.queue_time_us":153,"wal-append.run_cpu_time_us":331,"wal-append.run_wall_time_us":1000406}
I20240502 13:59:49.855443 24608 catalog_manager.cc:5484] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 reported cstate change: config changed from index -1 to 4, VOTER b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:49.871904 24884 consensus_queue.cc:237] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
W20240502 13:59:49.877329 24817 consensus_peers.cc:485] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf -> Peer 51d1006508074f10acd579729457b0c5 (127.22.55.131:42509): Couldn't send request to peer 51d1006508074f10acd579729457b0c5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27fa46c3e6214af2a67805b7dcfd9301. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:50.466421 26601 ts_tablet_manager.cc:902] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: Initiating tablet copy from peer 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:50.467259 26601 tablet_copy_client.cc:308] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:38051
I20240502 13:59:50.470889 24904 tablet_copy_service.cc:140] P 6e4fcab25c864479b96f38d291ebedaf: Received BeginTabletCopySession request for tablet 27fa46c3e6214af2a67805b7dcfd9301 from peer 51d1006508074f10acd579729457b0c5 ({username='slave'} at 127.22.55.131:53535)
I20240502 13:59:50.471030 24904 tablet_copy_service.cc:161] P 6e4fcab25c864479b96f38d291ebedaf: Beginning new tablet copy session on tablet 27fa46c3e6214af2a67805b7dcfd9301 from peer 51d1006508074f10acd579729457b0c5 at {username='slave'} at 127.22.55.131:53535: session id = 51d1006508074f10acd579729457b0c5-27fa46c3e6214af2a67805b7dcfd9301
I20240502 13:59:50.472297 24904 tablet_copy_source_session.cc:205] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:50.475710 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fa46c3e6214af2a67805b7dcfd9301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.480980 26601 tablet_copy_client.cc:783] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:50.481159 26601 tablet_copy_client.cc:647] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:50.482801 26601 tablet_copy_client.cc:523] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:50.484283 26601 tablet_bootstrap.cc:492] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: Bootstrap starting.
I20240502 13:59:50.489205 26601 log.cc:824] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:50.507863 26601 tablet_bootstrap.cc:492] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:50.508265 26601 tablet_bootstrap.cc:492] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: Bootstrap complete.
I20240502 13:59:50.508536 26601 ts_tablet_manager.cc:1367] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.005s
I20240502 13:59:50.510394 26601 raft_consensus.cc:348] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:50.510978 26601 raft_consensus.cc:729] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d1006508074f10acd579729457b0c5, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.511277 26601 consensus_queue.cc:260] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:50.512045 26601 ts_tablet_manager.cc:1398] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:50.512794 24904 tablet_copy_service.cc:342] P 6e4fcab25c864479b96f38d291ebedaf: Request end of tablet copy session 51d1006508074f10acd579729457b0c5-27fa46c3e6214af2a67805b7dcfd9301 received from {username='slave'} at 127.22.55.131:53535
I20240502 13:59:50.512907 24904 tablet_copy_service.cc:433] P 6e4fcab25c864479b96f38d291ebedaf: ending tablet copy session 51d1006508074f10acd579729457b0c5-27fa46c3e6214af2a67805b7dcfd9301 on tablet 27fa46c3e6214af2a67805b7dcfd9301 with peer 51d1006508074f10acd579729457b0c5
I20240502 13:59:51.091490 25887 raft_consensus.cc:1203] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:51.092828 26612 raft_consensus.cc:2932] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d1006508074f10acd579729457b0c5 (127.22.55.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:51.093441 25887 raft_consensus.cc:2932] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d1006508074f10acd579729457b0c5 (127.22.55.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:51.094506 26612 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43992 (request call id 0) took 1223 ms. Trace:
I20240502 13:59:51.094636 26612 rpcz_store.cc:276] 0502 13:59:49.871312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 13:59:49.871366 (+ 54us) service_pool.cc:225] Handling call
0502 13:59:49.872735 (+ 1369us) log.cc:842] Serialized 260 byte log entry
0502 13:59:51.094490 (+1221755us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":53,"raft.queue_time_us":104,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":274,"spinlock_wait_cycles":48384,"thread_start_us":107,"threads_started":2,"wal-append.queue_time_us":134,"wal-append.run_cpu_time_us":276,"wal-append.run_wall_time_us":1000339}
I20240502 13:59:51.096051 24608 catalog_manager.cc:5484] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 reported cstate change: config changed from index -1 to 4, VOTER 51d1006508074f10acd579729457b0c5 (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:51.112831 24753 consensus_queue.cc:237] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
W20240502 13:59:51.117343 24688 consensus_peers.cc:485] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 -> Peer 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132:39445): Couldn't send request to peer 8aa9c677a2c84c5082c708ae767eb3a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b81678740ddc4fce9da067dc3d8dff8d. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:51.564144 26639 ts_tablet_manager.cc:902] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: Initiating tablet copy from peer 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:51.564867 26639 tablet_copy_client.cc:308] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:36649
I20240502 13:59:51.568729 24773 tablet_copy_service.cc:140] P 7b4b1969054146119af3a958ef85de55: Received BeginTabletCopySession request for tablet b81678740ddc4fce9da067dc3d8dff8d from peer 8aa9c677a2c84c5082c708ae767eb3a8 ({username='slave'} at 127.22.55.132:42895)
I20240502 13:59:51.568851 24773 tablet_copy_service.cc:161] P 7b4b1969054146119af3a958ef85de55: Beginning new tablet copy session on tablet b81678740ddc4fce9da067dc3d8dff8d from peer 8aa9c677a2c84c5082c708ae767eb3a8 at {username='slave'} at 127.22.55.132:42895: session id = 8aa9c677a2c84c5082c708ae767eb3a8-b81678740ddc4fce9da067dc3d8dff8d
I20240502 13:59:51.569589 24773 tablet_copy_source_session.cc:205] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:51.572679 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81678740ddc4fce9da067dc3d8dff8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.577929 26639 tablet_copy_client.cc:783] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:51.578105 26639 tablet_copy_client.cc:647] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:51.579525 26639 tablet_copy_client.cc:523] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:51.580861 26639 tablet_bootstrap.cc:492] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: Bootstrap starting.
I20240502 13:59:51.585906 26639 log.cc:824] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:51.603668 26639 tablet_bootstrap.cc:492] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:51.604048 26639 tablet_bootstrap.cc:492] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: Bootstrap complete.
I20240502 13:59:51.604305 26639 ts_tablet_manager.cc:1367] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240502 13:59:51.606163 26639 raft_consensus.cc:348] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:51.606778 26639 raft_consensus.cc:729] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa9c677a2c84c5082c708ae767eb3a8, State: Initialized, Role: FOLLOWER
I20240502 13:59:51.607095 26639 consensus_queue.cc:260] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:51.607865 26639 ts_tablet_manager.cc:1398] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:51.608595 24773 tablet_copy_service.cc:342] P 7b4b1969054146119af3a958ef85de55: Request end of tablet copy session 8aa9c677a2c84c5082c708ae767eb3a8-b81678740ddc4fce9da067dc3d8dff8d received from {username='slave'} at 127.22.55.132:42895
I20240502 13:59:51.608688 24773 tablet_copy_service.cc:433] P 7b4b1969054146119af3a958ef85de55: ending tablet copy session 8aa9c677a2c84c5082c708ae767eb3a8-b81678740ddc4fce9da067dc3d8dff8d on tablet b81678740ddc4fce9da067dc3d8dff8d with peer 8aa9c677a2c84c5082c708ae767eb3a8
I20240502 13:59:52.073424 26018 raft_consensus.cc:1203] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:52.074574 26649 raft_consensus.cc:2932] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:52.075182 26018 raft_consensus.cc:2932] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:52.077801 24608 catalog_manager.cc:5484] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 reported cstate change: config changed from index -1 to 4, VOTER 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:52.098634 24884 consensus_queue.cc:237] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
W20240502 13:59:52.103299 24817 consensus_peers.cc:485] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf -> Peer a9e66f5a1b7f4202860a428d70203609 (127.22.55.135:41681): Couldn't send request to peer a9e66f5a1b7f4202860a428d70203609. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cae053fe22fd451e88e8e01eb93b76c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:52.652944 26674 ts_tablet_manager.cc:902] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: Initiating tablet copy from peer 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:52.653676 26674 tablet_copy_client.cc:308] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:38051
I20240502 13:59:52.657022 24904 tablet_copy_service.cc:140] P 6e4fcab25c864479b96f38d291ebedaf: Received BeginTabletCopySession request for tablet cae053fe22fd451e88e8e01eb93b76c5 from peer a9e66f5a1b7f4202860a428d70203609 ({username='slave'} at 127.22.55.135:60579)
I20240502 13:59:52.657119 24904 tablet_copy_service.cc:161] P 6e4fcab25c864479b96f38d291ebedaf: Beginning new tablet copy session on tablet cae053fe22fd451e88e8e01eb93b76c5 from peer a9e66f5a1b7f4202860a428d70203609 at {username='slave'} at 127.22.55.135:60579: session id = a9e66f5a1b7f4202860a428d70203609-cae053fe22fd451e88e8e01eb93b76c5
I20240502 13:59:52.657838 24904 tablet_copy_source_session.cc:205] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:52.660701 26674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae053fe22fd451e88e8e01eb93b76c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:52.665675 26674 tablet_copy_client.cc:783] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:52.665876 26674 tablet_copy_client.cc:647] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:52.667271 26674 tablet_copy_client.cc:523] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:52.668633 26674 tablet_bootstrap.cc:492] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: Bootstrap starting.
I20240502 13:59:52.675174 26674 log.cc:824] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:52.692881 26674 tablet_bootstrap.cc:492] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:52.693266 26674 tablet_bootstrap.cc:492] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: Bootstrap complete.
I20240502 13:59:52.693517 26674 ts_tablet_manager.cc:1367] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20240502 13:59:52.695396 26674 raft_consensus.cc:348] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:52.695930 26674 raft_consensus.cc:729] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e66f5a1b7f4202860a428d70203609, State: Initialized, Role: FOLLOWER
I20240502 13:59:52.696204 26674 consensus_queue.cc:260] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:52.696974 26674 ts_tablet_manager.cc:1398] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240502 13:59:52.697703 24904 tablet_copy_service.cc:342] P 6e4fcab25c864479b96f38d291ebedaf: Request end of tablet copy session a9e66f5a1b7f4202860a428d70203609-cae053fe22fd451e88e8e01eb93b76c5 received from {username='slave'} at 127.22.55.135:60579
I20240502 13:59:52.697793 24904 tablet_copy_service.cc:433] P 6e4fcab25c864479b96f38d291ebedaf: ending tablet copy session a9e66f5a1b7f4202860a428d70203609-cae053fe22fd451e88e8e01eb93b76c5 on tablet cae053fe22fd451e88e8e01eb93b76c5 with peer a9e66f5a1b7f4202860a428d70203609
I20240502 13:59:53.051586 26411 raft_consensus.cc:1203] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:53.052775 26683 raft_consensus.cc:2932] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a9e66f5a1b7f4202860a428d70203609 (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:53.053287 26411 raft_consensus.cc:2932] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a9e66f5a1b7f4202860a428d70203609 (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:53.056243 24610 catalog_manager.cc:5484] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: config changed from index -1 to 4, VOTER a9e66f5a1b7f4202860a428d70203609 (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:53.071722 24753 consensus_queue.cc:237] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
W20240502 13:59:53.076231 24688 consensus_peers.cc:485] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 -> Peer c970e31aa29e4944ab5fb3024984aefd (127.22.55.134:41009): Couldn't send request to peer c970e31aa29e4944ab5fb3024984aefd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f943154c90654cd39f602a0fc2b90ceb. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:53.496204 26708 ts_tablet_manager.cc:902] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: Initiating tablet copy from peer 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:53.496899 26708 tablet_copy_client.cc:308] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:36649
I20240502 13:59:53.500429 24773 tablet_copy_service.cc:140] P 7b4b1969054146119af3a958ef85de55: Received BeginTabletCopySession request for tablet f943154c90654cd39f602a0fc2b90ceb from peer c970e31aa29e4944ab5fb3024984aefd ({username='slave'} at 127.22.55.134:56997)
I20240502 13:59:53.500540 24773 tablet_copy_service.cc:161] P 7b4b1969054146119af3a958ef85de55: Beginning new tablet copy session on tablet f943154c90654cd39f602a0fc2b90ceb from peer c970e31aa29e4944ab5fb3024984aefd at {username='slave'} at 127.22.55.134:56997: session id = c970e31aa29e4944ab5fb3024984aefd-f943154c90654cd39f602a0fc2b90ceb
I20240502 13:59:53.501348 24773 tablet_copy_source_session.cc:205] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:53.504244 26708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f943154c90654cd39f602a0fc2b90ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:53.509204 26708 tablet_copy_client.cc:783] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:53.509377 26708 tablet_copy_client.cc:647] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:53.510804 26708 tablet_copy_client.cc:523] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:53.512287 26708 tablet_bootstrap.cc:492] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: Bootstrap starting.
I20240502 13:59:53.517822 26708 log.cc:824] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:53.536254 26708 tablet_bootstrap.cc:492] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:53.536607 26708 tablet_bootstrap.cc:492] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: Bootstrap complete.
I20240502 13:59:53.536857 26708 ts_tablet_manager.cc:1367] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.001s
I20240502 13:59:53.538800 26708 raft_consensus.cc:348] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:53.539489 26708 raft_consensus.cc:729] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c970e31aa29e4944ab5fb3024984aefd, State: Initialized, Role: FOLLOWER
I20240502 13:59:53.539765 26708 consensus_queue.cc:260] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 13:59:53.540560 26708 ts_tablet_manager.cc:1398] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 13:59:53.541255 24773 tablet_copy_service.cc:342] P 7b4b1969054146119af3a958ef85de55: Request end of tablet copy session c970e31aa29e4944ab5fb3024984aefd-f943154c90654cd39f602a0fc2b90ceb received from {username='slave'} at 127.22.55.134:56997
I20240502 13:59:53.541342 24773 tablet_copy_service.cc:433] P 7b4b1969054146119af3a958ef85de55: ending tablet copy session c970e31aa29e4944ab5fb3024984aefd-f943154c90654cd39f602a0fc2b90ceb on tablet f943154c90654cd39f602a0fc2b90ceb with peer c970e31aa29e4944ab5fb3024984aefd
I20240502 13:59:53.923470 26280 raft_consensus.cc:1203] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:53.924573 26716 raft_consensus.cc:2932] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c970e31aa29e4944ab5fb3024984aefd (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 13:59:53.925103 26280 raft_consensus.cc:2932] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c970e31aa29e4944ab5fb3024984aefd (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 13:59:53.926918 24610 catalog_manager.cc:5484] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 reported cstate change: config changed from index -1 to 4, VOTER c970e31aa29e4944ab5fb3024984aefd (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 13:59:53.944450 24884 consensus_queue.cc:237] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
W20240502 13:59:53.948217 24817 consensus_peers.cc:485] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf -> Peer b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133:38561): Couldn't send request to peer b2a245b6dbf34e03a431a50a7b54ce92. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8c1d90d5d8449aac5e3e1e81d54523. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:54.455879 26741 ts_tablet_manager.cc:902] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: Initiating tablet copy from peer 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:54.456359 26741 tablet_copy_client.cc:308] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:38051
I20240502 13:59:54.459530 24904 tablet_copy_service.cc:140] P 6e4fcab25c864479b96f38d291ebedaf: Received BeginTabletCopySession request for tablet ff8c1d90d5d8449aac5e3e1e81d54523 from peer b2a245b6dbf34e03a431a50a7b54ce92 ({username='slave'} at 127.22.55.133:59411)
I20240502 13:59:54.459626 24904 tablet_copy_service.cc:161] P 6e4fcab25c864479b96f38d291ebedaf: Beginning new tablet copy session on tablet ff8c1d90d5d8449aac5e3e1e81d54523 from peer b2a245b6dbf34e03a431a50a7b54ce92 at {username='slave'} at 127.22.55.133:59411: session id = b2a245b6dbf34e03a431a50a7b54ce92-ff8c1d90d5d8449aac5e3e1e81d54523
I20240502 13:59:54.460350 24904 tablet_copy_source_session.cc:205] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:54.461138 26741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8c1d90d5d8449aac5e3e1e81d54523. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:54.462910 26741 tablet_copy_client.cc:783] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:54.463066 26741 tablet_copy_client.cc:647] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:54.464100 26741 tablet_copy_client.cc:523] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:54.465071 26741 tablet_bootstrap.cc:492] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: Bootstrap starting.
I20240502 13:59:54.482740 26741 tablet_bootstrap.cc:492] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:54.483052 26741 tablet_bootstrap.cc:492] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: Bootstrap complete.
I20240502 13:59:54.483229 26741 ts_tablet_manager.cc:1367] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.005s
I20240502 13:59:54.483708 26741 raft_consensus.cc:348] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:54.483938 26741 raft_consensus.cc:729] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a245b6dbf34e03a431a50a7b54ce92, State: Initialized, Role: FOLLOWER
I20240502 13:59:54.484043 26741 consensus_queue.cc:260] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:54.484388 26741 ts_tablet_manager.cc:1398] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:54.488734 24904 tablet_copy_service.cc:342] P 6e4fcab25c864479b96f38d291ebedaf: Request end of tablet copy session b2a245b6dbf34e03a431a50a7b54ce92-ff8c1d90d5d8449aac5e3e1e81d54523 received from {username='slave'} at 127.22.55.133:59411
I20240502 13:59:54.488822 24904 tablet_copy_service.cc:433] P 6e4fcab25c864479b96f38d291ebedaf: ending tablet copy session b2a245b6dbf34e03a431a50a7b54ce92-ff8c1d90d5d8449aac5e3e1e81d54523 on tablet ff8c1d90d5d8449aac5e3e1e81d54523 with peer b2a245b6dbf34e03a431a50a7b54ce92
I20240502 13:59:54.848731 26149 raft_consensus.cc:1203] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:54.849601 26750 raft_consensus.cc:2932] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:54.850024 26149 raft_consensus.cc:2932] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:54.853257 24608 catalog_manager.cc:5484] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: config changed from index -1 to 4, VOTER b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:54.868974 24753 consensus_queue.cc:237] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
W20240502 13:59:54.873283 24686 consensus_peers.cc:485] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 -> Peer 51d1006508074f10acd579729457b0c5 (127.22.55.131:42509): Couldn't send request to peer 51d1006508074f10acd579729457b0c5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94738674b9af470f922b593daec77442. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:55.257189 26775 ts_tablet_manager.cc:902] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: Initiating tablet copy from peer 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:55.257695 26775 tablet_copy_client.cc:308] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:36649
I20240502 13:59:55.261091 24773 tablet_copy_service.cc:140] P 7b4b1969054146119af3a958ef85de55: Received BeginTabletCopySession request for tablet 94738674b9af470f922b593daec77442 from peer 51d1006508074f10acd579729457b0c5 ({username='slave'} at 127.22.55.131:57037)
I20240502 13:59:55.261204 24773 tablet_copy_service.cc:161] P 7b4b1969054146119af3a958ef85de55: Beginning new tablet copy session on tablet 94738674b9af470f922b593daec77442 from peer 51d1006508074f10acd579729457b0c5 at {username='slave'} at 127.22.55.131:57037: session id = 51d1006508074f10acd579729457b0c5-94738674b9af470f922b593daec77442
I20240502 13:59:55.261986 24773 tablet_copy_source_session.cc:205] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:55.262915 26775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94738674b9af470f922b593daec77442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:55.264760 26775 tablet_copy_client.cc:783] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:55.264892 26775 tablet_copy_client.cc:647] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:55.265962 26775 tablet_copy_client.cc:523] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:55.266966 26775 tablet_bootstrap.cc:492] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: Bootstrap starting.
I20240502 13:59:55.284137 26775 tablet_bootstrap.cc:492] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:55.284476 26775 tablet_bootstrap.cc:492] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: Bootstrap complete.
I20240502 13:59:55.284654 26775 ts_tablet_manager.cc:1367] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20240502 13:59:55.285144 26775 raft_consensus.cc:348] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:55.285353 26775 raft_consensus.cc:729] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d1006508074f10acd579729457b0c5, State: Initialized, Role: FOLLOWER
I20240502 13:59:55.285456 26775 consensus_queue.cc:260] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:55.285810 26775 ts_tablet_manager.cc:1398] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:55.286376 24773 tablet_copy_service.cc:342] P 7b4b1969054146119af3a958ef85de55: Request end of tablet copy session 51d1006508074f10acd579729457b0c5-94738674b9af470f922b593daec77442 received from {username='slave'} at 127.22.55.131:57037
I20240502 13:59:55.286451 24773 tablet_copy_service.cc:433] P 7b4b1969054146119af3a958ef85de55: ending tablet copy session 51d1006508074f10acd579729457b0c5-94738674b9af470f922b593daec77442 on tablet 94738674b9af470f922b593daec77442 with peer 51d1006508074f10acd579729457b0c5
I20240502 13:59:55.683027 25887 raft_consensus.cc:1203] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:55.683977 26783 raft_consensus.cc:2932] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d1006508074f10acd579729457b0c5 (127.22.55.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:55.684396 25887 raft_consensus.cc:2932] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 51d1006508074f10acd579729457b0c5 (127.22.55.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:55.706640 24608 catalog_manager.cc:5484] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 reported cstate change: config changed from index -1 to 4, VOTER 51d1006508074f10acd579729457b0c5 (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:55.724723 24884 consensus_queue.cc:237] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
W20240502 13:59:55.728690 24819 consensus_peers.cc:485] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf -> Peer 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132:39445): Couldn't send request to peer 8aa9c677a2c84c5082c708ae767eb3a8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94e5deba02294d4e85fd4bad586a78d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:56.237191 26808 ts_tablet_manager.cc:902] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: Initiating tablet copy from peer 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:56.237655 26808 tablet_copy_client.cc:308] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:38051
I20240502 13:59:56.241117 24904 tablet_copy_service.cc:140] P 6e4fcab25c864479b96f38d291ebedaf: Received BeginTabletCopySession request for tablet 94e5deba02294d4e85fd4bad586a78d5 from peer 8aa9c677a2c84c5082c708ae767eb3a8 ({username='slave'} at 127.22.55.132:55915)
I20240502 13:59:56.241212 24904 tablet_copy_service.cc:161] P 6e4fcab25c864479b96f38d291ebedaf: Beginning new tablet copy session on tablet 94e5deba02294d4e85fd4bad586a78d5 from peer 8aa9c677a2c84c5082c708ae767eb3a8 at {username='slave'} at 127.22.55.132:55915: session id = 8aa9c677a2c84c5082c708ae767eb3a8-94e5deba02294d4e85fd4bad586a78d5
I20240502 13:59:56.241895 24904 tablet_copy_source_session.cc:205] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:56.242638 26808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e5deba02294d4e85fd4bad586a78d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:56.244457 26808 tablet_copy_client.cc:783] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:56.244639 26808 tablet_copy_client.cc:647] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:56.245673 26808 tablet_copy_client.cc:523] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:56.246618 26808 tablet_bootstrap.cc:492] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: Bootstrap starting.
I20240502 13:59:56.263269 26808 tablet_bootstrap.cc:492] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:56.263532 26808 tablet_bootstrap.cc:492] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: Bootstrap complete.
I20240502 13:59:56.263727 26808 ts_tablet_manager.cc:1367] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240502 13:59:56.264235 26808 raft_consensus.cc:348] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 13:59:56.264444 26808 raft_consensus.cc:729] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa9c677a2c84c5082c708ae767eb3a8, State: Initialized, Role: FOLLOWER
I20240502 13:59:56.264552 26808 consensus_queue.cc:260] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:56.264901 26808 ts_tablet_manager.cc:1398] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:56.265504 24904 tablet_copy_service.cc:342] P 6e4fcab25c864479b96f38d291ebedaf: Request end of tablet copy session 8aa9c677a2c84c5082c708ae767eb3a8-94e5deba02294d4e85fd4bad586a78d5 received from {username='slave'} at 127.22.55.132:55915
I20240502 13:59:56.265586 24904 tablet_copy_service.cc:433] P 6e4fcab25c864479b96f38d291ebedaf: ending tablet copy session 8aa9c677a2c84c5082c708ae767eb3a8-94e5deba02294d4e85fd4bad586a78d5 on tablet 94e5deba02294d4e85fd4bad586a78d5 with peer 8aa9c677a2c84c5082c708ae767eb3a8
I20240502 13:59:56.739267 26018 raft_consensus.cc:1203] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:56.740083 26817 raft_consensus.cc:2932] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:56.740512 26018 raft_consensus.cc:2932] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:56.741441 26817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33358 (request call id 0) took 1017 ms. Trace:
I20240502 13:59:56.741549 26817 rpcz_store.cc:276] 0502 13:59:55.724385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 13:59:55.724443 (+ 58us) service_pool.cc:225] Handling call
0502 13:59:55.725168 (+ 725us) log.cc:842] Serialized 260 byte log entry
0502 13:59:56.741424 (+1016256us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":39,"raft.queue_time_us":41,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":124,"wal-append.run_cpu_time_us":333,"wal-append.run_wall_time_us":1000487}
I20240502 13:59:56.743606 24603 catalog_manager.cc:5484] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 reported cstate change: config changed from index -1 to 4, VOTER 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:56.759558 24753 consensus_queue.cc:237] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
W20240502 13:59:56.763559 24686 consensus_peers.cc:485] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 -> Peer a9e66f5a1b7f4202860a428d70203609 (127.22.55.135:41681): Couldn't send request to peer a9e66f5a1b7f4202860a428d70203609. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ae2eb2957c749b4a34b07944a5c3592. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:57.343773 26844 ts_tablet_manager.cc:902] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: Initiating tablet copy from peer 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:57.344308 26844 tablet_copy_client.cc:308] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:36649
I20240502 13:59:57.347602 24773 tablet_copy_service.cc:140] P 7b4b1969054146119af3a958ef85de55: Received BeginTabletCopySession request for tablet 1ae2eb2957c749b4a34b07944a5c3592 from peer a9e66f5a1b7f4202860a428d70203609 ({username='slave'} at 127.22.55.135:49631)
I20240502 13:59:57.347694 24773 tablet_copy_service.cc:161] P 7b4b1969054146119af3a958ef85de55: Beginning new tablet copy session on tablet 1ae2eb2957c749b4a34b07944a5c3592 from peer a9e66f5a1b7f4202860a428d70203609 at {username='slave'} at 127.22.55.135:49631: session id = a9e66f5a1b7f4202860a428d70203609-1ae2eb2957c749b4a34b07944a5c3592
I20240502 13:59:57.363700 24773 tablet_copy_source_session.cc:205] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:57.364694 26844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae2eb2957c749b4a34b07944a5c3592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:57.366469 26844 tablet_copy_client.cc:783] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:57.366617 26844 tablet_copy_client.cc:647] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:57.367771 26844 tablet_copy_client.cc:523] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:57.368736 26844 tablet_bootstrap.cc:492] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: Bootstrap starting.
I20240502 13:59:57.386251 26844 tablet_bootstrap.cc:492] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 13:59:57.386507 26844 tablet_bootstrap.cc:492] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: Bootstrap complete.
I20240502 13:59:57.386641 26844 ts_tablet_manager.cc:1367] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240502 13:59:57.387193 26844 raft_consensus.cc:348] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 13:59:57.387380 26844 raft_consensus.cc:729] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e66f5a1b7f4202860a428d70203609, State: Initialized, Role: FOLLOWER
I20240502 13:59:57.387468 26844 consensus_queue.cc:260] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:57.387768 26844 ts_tablet_manager.cc:1398] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240502 13:59:57.388327 24773 tablet_copy_service.cc:342] P 7b4b1969054146119af3a958ef85de55: Request end of tablet copy session a9e66f5a1b7f4202860a428d70203609-1ae2eb2957c749b4a34b07944a5c3592 received from {username='slave'} at 127.22.55.135:49631
I20240502 13:59:57.388408 24773 tablet_copy_service.cc:433] P 7b4b1969054146119af3a958ef85de55: ending tablet copy session a9e66f5a1b7f4202860a428d70203609-1ae2eb2957c749b4a34b07944a5c3592 on tablet 1ae2eb2957c749b4a34b07944a5c3592 with peer a9e66f5a1b7f4202860a428d70203609
I20240502 13:59:57.847400 26411 raft_consensus.cc:1203] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 13:59:57.848275 26853 raft_consensus.cc:2932] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a9e66f5a1b7f4202860a428d70203609 (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:57.848605 26411 raft_consensus.cc:2932] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a9e66f5a1b7f4202860a428d70203609 (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:57.849596 26853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:48116 (request call id 0) took 1090 ms. Trace:
I20240502 13:59:57.849656 26853 rpcz_store.cc:276] 0502 13:59:56.759283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 13:59:56.759329 (+ 46us) service_pool.cc:225] Handling call
0502 13:59:56.759952 (+ 623us) log.cc:842] Serialized 260 byte log entry
0502 13:59:57.849579 (+1089627us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":46,"raft.queue_time_us":39,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":120,"wal-append.run_cpu_time_us":303,"wal-append.run_wall_time_us":1000369}
I20240502 13:59:57.852205 24603 catalog_manager.cc:5484] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 reported cstate change: config changed from index -1 to 4, VOTER a9e66f5a1b7f4202860a428d70203609 (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:57.913496 24732 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:57.913702 25867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:57.917764 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:57.920284 26391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:57.921993 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:57.927008 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:57.941905 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:57.971670 24753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ae2eb2957c749b4a34b07944a5c3592"
dest_uuid: "7b4b1969054146119af3a958ef85de55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48130
I20240502 13:59:57.971841 24753 raft_consensus.cc:595] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:58.022799 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.023289 24732 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:58.024077 25867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.025022 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.032761 26391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.038722 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.044095 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.129128 25867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.134461 24732 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:58.136802 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.137411 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.137862 26391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.143236 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.159366 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.182010 24753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94738674b9af470f922b593daec77442"
dest_uuid: "7b4b1969054146119af3a958ef85de55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48168
I20240502 13:59:58.182166 24753 raft_consensus.cc:595] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:58.232514 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.239506 24732 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:58.240764 26391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.242085 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.243115 25867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.243295 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.274000 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.342996 25867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.349789 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.351051 24732 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:58.355195 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.357259 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.363046 26391 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.373569 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.388402 27054 raft_consensus.cc:979] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: : Instructing follower a9e66f5a1b7f4202860a428d70203609 to start an election
I20240502 13:59:58.388526 27054 raft_consensus.cc:1067] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Signalling peer a9e66f5a1b7f4202860a428d70203609 to start an election
I20240502 13:59:58.389142 26411 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ae2eb2957c749b4a34b07944a5c3592"
dest_uuid: "a9e66f5a1b7f4202860a428d70203609"
from {username='slave'} at 127.22.55.129:53075
I20240502 13:59:58.389328 26411 raft_consensus.cc:482] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:58.389427 26411 raft_consensus.cc:3037] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.390241 26411 raft_consensus.cc:504] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:58.391096 26411 leader_election.cc:288] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [CANDIDATE]: Term 2 election: Requested vote from peers 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:58.391496 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae2eb2957c749b4a34b07944a5c3592" candidate_uuid: "a9e66f5a1b7f4202860a428d70203609" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7b4b1969054146119af3a958ef85de55"
I20240502 13:59:58.391664 24753 raft_consensus.cc:3032] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:58.391706 24753 raft_consensus.cc:729] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:58.391844 24753 consensus_queue.cc:260] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:58.392092 24753 raft_consensus.cc:3037] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.392859 24753 raft_consensus.cc:2454] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e66f5a1b7f4202860a428d70203609 in term 2.
I20240502 13:59:58.393211 26345 leader_election.cc:302] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55, a9e66f5a1b7f4202860a428d70203609; no voters:
I20240502 13:59:58.393555 27055 raft_consensus.cc:2781] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:58.393744 27055 raft_consensus.cc:686] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 2 LEADER]: Becoming Leader. State: Replica: a9e66f5a1b7f4202860a428d70203609, State: Running, Role: LEADER
I20240502 13:59:58.394098 27055 consensus_queue.cc:237] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:58.397028 24603 catalog_manager.cc:5484] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 reported cstate change: term changed from 1 to 2, leader changed from 7b4b1969054146119af3a958ef85de55 (127.22.55.129) to a9e66f5a1b7f4202860a428d70203609 (127.22.55.135). New cstate: current_term: 2 leader_uuid: "a9e66f5a1b7f4202860a428d70203609" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:58.400902 24884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27fa46c3e6214af2a67805b7dcfd9301"
dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33400
I20240502 13:59:58.401077 24884 raft_consensus.cc:595] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:58.414708 26853 raft_consensus.cc:979] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: : Instructing follower 51d1006508074f10acd579729457b0c5 to start an election
I20240502 13:59:58.414916 26853 raft_consensus.cc:1067] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Signalling peer 51d1006508074f10acd579729457b0c5 to start an election
I20240502 13:59:58.415513 25887 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94738674b9af470f922b593daec77442"
dest_uuid: "51d1006508074f10acd579729457b0c5"
from {username='slave'} at 127.22.55.129:56077
I20240502 13:59:58.415714 25887 raft_consensus.cc:482] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:58.415831 25887 raft_consensus.cc:3037] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.416781 25887 raft_consensus.cc:504] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:58.417675 25887 leader_election.cc:288] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [CANDIDATE]: Term 2 election: Requested vote from peers 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:58.418056 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94738674b9af470f922b593daec77442" candidate_uuid: "51d1006508074f10acd579729457b0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7b4b1969054146119af3a958ef85de55"
I20240502 13:59:58.418174 24753 raft_consensus.cc:3032] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:58.418229 24753 raft_consensus.cc:729] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:58.418432 24753 consensus_queue.cc:260] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:58.418573 24753 raft_consensus.cc:3037] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.419497 24753 raft_consensus.cc:2454] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d1006508074f10acd579729457b0c5 in term 2.
I20240502 13:59:58.419871 25821 leader_election.cc:302] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 51d1006508074f10acd579729457b0c5, 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 13:59:58.420257 27071 raft_consensus.cc:2781] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:58.420428 27071 raft_consensus.cc:686] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 2 LEADER]: Becoming Leader. State: Replica: 51d1006508074f10acd579729457b0c5, State: Running, Role: LEADER
I20240502 13:59:58.420754 27071 consensus_queue.cc:237] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:58.423388 24603 catalog_manager.cc:5484] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 reported cstate change: term changed from 1 to 2, leader changed from 7b4b1969054146119af3a958ef85de55 (127.22.55.129) to 51d1006508074f10acd579729457b0c5 (127.22.55.131). New cstate: current_term: 2 leader_uuid: "51d1006508074f10acd579729457b0c5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:58.450240 25867 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.453822 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.461747 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.462020 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.462215 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.462443 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.471329 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.559546 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.560475 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.560803 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.564409 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.571480 25867 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.572579 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.582556 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.613822 24884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff8c1d90d5d8449aac5e3e1e81d54523"
dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33436
I20240502 13:59:58.613978 24884 raft_consensus.cc:595] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:58.663620 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.666395 25867 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.667397 26129 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.668395 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.677340 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.689389 24863 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.694145 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.743350 26817 raft_consensus.cc:979] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: : Instructing follower 51d1006508074f10acd579729457b0c5 to start an election
I20240502 13:59:58.743481 26817 raft_consensus.cc:1067] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Signalling peer 51d1006508074f10acd579729457b0c5 to start an election
I20240502 13:59:58.744071 25887 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27fa46c3e6214af2a67805b7dcfd9301"
dest_uuid: "51d1006508074f10acd579729457b0c5"
from {username='slave'} at 127.22.55.130:35901
I20240502 13:59:58.744226 25887 raft_consensus.cc:482] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:58.744277 25887 raft_consensus.cc:3037] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.745214 25887 raft_consensus.cc:504] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:58.745535 25887 leader_election.cc:288] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [CANDIDATE]: Term 2 election: Requested vote from peers 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:58.746186 24884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fa46c3e6214af2a67805b7dcfd9301" candidate_uuid: "51d1006508074f10acd579729457b0c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
I20240502 13:59:58.746361 24884 raft_consensus.cc:3032] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:58.746408 24884 raft_consensus.cc:729] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:58.746588 24884 consensus_queue.cc:260] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:58.746882 24884 raft_consensus.cc:3037] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.747822 24884 raft_consensus.cc:2454] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51d1006508074f10acd579729457b0c5 in term 2.
I20240502 13:59:58.748170 25820 leader_election.cc:302] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 51d1006508074f10acd579729457b0c5, 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 13:59:58.748432 27071 raft_consensus.cc:2781] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:58.748528 27071 raft_consensus.cc:686] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 2 LEADER]: Becoming Leader. State: Replica: 51d1006508074f10acd579729457b0c5, State: Running, Role: LEADER
I20240502 13:59:58.748672 27071 consensus_queue.cc:237] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 13:59:58.750209 24608 catalog_manager.cc:5484] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 reported cstate change: term changed from 1 to 2, leader changed from 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) to 51d1006508074f10acd579729457b0c5 (127.22.55.131). New cstate: current_term: 2 leader_uuid: "51d1006508074f10acd579729457b0c5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 13:59:58.758431 27190 raft_consensus.cc:979] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: : Instructing follower b2a245b6dbf34e03a431a50a7b54ce92 to start an election
I20240502 13:59:58.758543 27190 raft_consensus.cc:1067] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Signalling peer b2a245b6dbf34e03a431a50a7b54ce92 to start an election
I20240502 13:59:58.759071 26149 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff8c1d90d5d8449aac5e3e1e81d54523"
dest_uuid: "b2a245b6dbf34e03a431a50a7b54ce92"
from {username='slave'} at 127.22.55.130:41827
I20240502 13:59:58.759238 26149 raft_consensus.cc:482] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:58.759347 26149 raft_consensus.cc:3037] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.760468 26149 raft_consensus.cc:504] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:58.761448 26149 leader_election.cc:288] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [CANDIDATE]: Term 2 election: Requested vote from peers 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:58.762653 24884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8c1d90d5d8449aac5e3e1e81d54523" candidate_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
I20240502 13:59:58.762753 24884 raft_consensus.cc:3032] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:58.762781 24884 raft_consensus.cc:729] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:58.762952 24884 consensus_queue.cc:260] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:58.763072 24884 raft_consensus.cc:3037] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:58.764005 24884 raft_consensus.cc:2454] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a245b6dbf34e03a431a50a7b54ce92 in term 2.
I20240502 13:59:58.764446 26082 leader_election.cc:302] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf, b2a245b6dbf34e03a431a50a7b54ce92; no voters:
I20240502 13:59:58.765290 27203 raft_consensus.cc:2781] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:58.765499 27203 raft_consensus.cc:686] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 LEADER]: Becoming Leader. State: Replica: b2a245b6dbf34e03a431a50a7b54ce92, State: Running, Role: LEADER
I20240502 13:59:58.765864 27203 consensus_queue.cc:237] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:58.768215 24608 catalog_manager.cc:5484] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 reported cstate change: term changed from 1 to 2, leader changed from 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) to b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133). New cstate: current_term: 2 leader_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:58.781455 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:58.781920 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.783851 24863 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:58.784212 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.784967 26129 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.791731 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.823732 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.844468 24753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d465839425704eef962eb345979233f6"
dest_uuid: "7b4b1969054146119af3a958ef85de55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48228
I20240502 13:59:58.844620 24753 raft_consensus.cc:595] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:58.847716 24753 raft_consensus.cc:1261] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Refusing update from remote peer 51d1006508074f10acd579729457b0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 13:59:58.848459 27071 consensus_queue.cc:1035] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:58.898245 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:58.899438 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:58.901733 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.907500 24863 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:58.910902 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:58.911787 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.919246 26129 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:58.998643 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.001963 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.006790 26129 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.007723 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:59.011617 24753 raft_consensus.cc:1261] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Refusing update from remote peer a9e66f5a1b7f4202860a428d70203609: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 13:59:59.012208 27293 consensus_queue.cc:1035] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:59.014454 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.023380 24863 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:59.025169 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.055665 24884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94e5deba02294d4e85fd4bad586a78d5"
dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33502
I20240502 13:59:59.055799 24884 raft_consensus.cc:595] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:59.108049 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.109930 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.110184 24863 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:59.110447 26129 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.111423 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.112591 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:59.137167 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.141913 24884 raft_consensus.cc:1261] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Refusing update from remote peer 51d1006508074f10acd579729457b0c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 13:59:59.142539 27072 consensus_queue.cc:1035] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:59.214048 24884 raft_consensus.cc:1261] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Refusing update from remote peer b2a245b6dbf34e03a431a50a7b54ce92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 13:59:59.214599 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.215137 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:59.215173 27203 consensus_queue.cc:1035] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:59.222785 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.223282 26129 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.224903 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.231688 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.232136 24863 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:59.270387 24753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b81678740ddc4fce9da067dc3d8dff8d"
dest_uuid: "7b4b1969054146119af3a958ef85de55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48276
I20240502 13:59:59.270537 24753 raft_consensus.cc:595] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:59.320003 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.323531 24732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:59.325445 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.327960 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.335618 26129 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.337064 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.338343 24863 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:59.375144 26853 raft_consensus.cc:979] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: : Instructing follower b2a245b6dbf34e03a431a50a7b54ce92 to start an election
I20240502 13:59:59.375264 26853 raft_consensus.cc:1067] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Signalling peer b2a245b6dbf34e03a431a50a7b54ce92 to start an election
I20240502 13:59:59.375699 26149 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d465839425704eef962eb345979233f6"
dest_uuid: "b2a245b6dbf34e03a431a50a7b54ce92"
from {username='slave'} at 127.22.55.129:35409
I20240502 13:59:59.375844 26149 raft_consensus.cc:482] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:59.375906 26149 raft_consensus.cc:3037] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.376987 26149 raft_consensus.cc:504] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:59.377388 26149 leader_election.cc:288] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [CANDIDATE]: Term 2 election: Requested vote from peers 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:59.377723 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d465839425704eef962eb345979233f6" candidate_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7b4b1969054146119af3a958ef85de55"
I20240502 13:59:59.377864 24753 raft_consensus.cc:3032] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:59.377918 24753 raft_consensus.cc:729] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:59.378057 24753 consensus_queue.cc:260] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:59.378211 24753 raft_consensus.cc:3037] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.379201 24753 raft_consensus.cc:2454] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a245b6dbf34e03a431a50a7b54ce92 in term 2.
I20240502 13:59:59.379523 26083 leader_election.cc:302] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55, b2a245b6dbf34e03a431a50a7b54ce92; no voters:
I20240502 13:59:59.379706 27204 raft_consensus.cc:2781] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:59.379797 27204 raft_consensus.cc:686] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 LEADER]: Becoming Leader. State: Replica: b2a245b6dbf34e03a431a50a7b54ce92, State: Running, Role: LEADER
I20240502 13:59:59.379927 27204 consensus_queue.cc:237] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 13:59:59.381327 24607 catalog_manager.cc:5484] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 reported cstate change: term changed from 1 to 2, leader changed from 7b4b1969054146119af3a958ef85de55 (127.22.55.129) to b2a245b6dbf34e03a431a50a7b54ce92 (127.22.55.133). New cstate: current_term: 2 leader_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 13:59:59.424479 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.426414 25998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.427356 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.427835 24732 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 13:59:59.433866 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.445838 24863 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:59.451741 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.454558 27190 raft_consensus.cc:979] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: : Instructing follower 8aa9c677a2c84c5082c708ae767eb3a8 to start an election
I20240502 13:59:59.454638 27190 raft_consensus.cc:1067] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Signalling peer 8aa9c677a2c84c5082c708ae767eb3a8 to start an election
I20240502 13:59:59.455096 26018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94e5deba02294d4e85fd4bad586a78d5"
dest_uuid: "8aa9c677a2c84c5082c708ae767eb3a8"
from {username='slave'} at 127.22.55.130:36119
I20240502 13:59:59.455286 26018 raft_consensus.cc:482] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:59.455377 26018 raft_consensus.cc:3037] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.456414 26018 raft_consensus.cc:504] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:59.457353 26018 leader_election.cc:288] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [CANDIDATE]: Term 2 election: Requested vote from peers 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:59.457798 24884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e5deba02294d4e85fd4bad586a78d5" candidate_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
I20240502 13:59:59.457888 24884 raft_consensus.cc:3032] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:59.457914 24884 raft_consensus.cc:729] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:59.458032 24884 consensus_queue.cc:260] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:59.458099 24884 raft_consensus.cc:3037] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.458909 24884 raft_consensus.cc:2454] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aa9c677a2c84c5082c708ae767eb3a8 in term 2.
I20240502 13:59:59.463023 25951 leader_election.cc:302] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf, 8aa9c677a2c84c5082c708ae767eb3a8; no voters:
I20240502 13:59:59.463373 27456 raft_consensus.cc:2781] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:59.463528 27456 raft_consensus.cc:686] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 LEADER]: Becoming Leader. State: Replica: 8aa9c677a2c84c5082c708ae767eb3a8, State: Running, Role: LEADER
I20240502 13:59:59.463852 27456 consensus_queue.cc:237] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:59.466328 24603 catalog_manager.cc:5484] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 reported cstate change: term changed from 1 to 2, leader changed from 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) to 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:59.480432 24884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cae053fe22fd451e88e8e01eb93b76c5"
dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33558
I20240502 13:59:59.480581 24884 raft_consensus.cc:595] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:59.533062 24863 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 13:59:59.534638 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.536782 24732 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 13:59:59.540920 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.543408 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.549944 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.550149 25998 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.638132 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.638686 24863 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 13:59:59.639196 25998 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.642786 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.647148 24732 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 13:59:59.655246 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.663849 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.688961 24753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f943154c90654cd39f602a0fc2b90ceb"
dest_uuid: "7b4b1969054146119af3a958ef85de55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48338
I20240502 13:59:59.689097 24753 raft_consensus.cc:595] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Received request to transfer leadership
I20240502 13:59:59.737111 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.740258 24732 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 13:59:59.741127 25998 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.746533 26391 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:59.749975 24863 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 13:59:59.751506 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:59.752847 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:59.776150 27424 raft_consensus.cc:979] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: : Instructing follower 8aa9c677a2c84c5082c708ae767eb3a8 to start an election
I20240502 13:59:59.776273 26853 raft_consensus.cc:1067] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Signalling peer 8aa9c677a2c84c5082c708ae767eb3a8 to start an election
I20240502 13:59:59.776819 26018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b81678740ddc4fce9da067dc3d8dff8d"
dest_uuid: "8aa9c677a2c84c5082c708ae767eb3a8"
from {username='slave'} at 127.22.55.129:45943
I20240502 13:59:59.776952 26018 raft_consensus.cc:482] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:59.777019 26018 raft_consensus.cc:3037] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.778067 26018 raft_consensus.cc:504] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:59.778436 26018 leader_election.cc:288] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [CANDIDATE]: Term 2 election: Requested vote from peers 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:59.778950 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81678740ddc4fce9da067dc3d8dff8d" candidate_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7b4b1969054146119af3a958ef85de55"
I20240502 13:59:59.779070 24753 raft_consensus.cc:3032] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:59.779124 24753 raft_consensus.cc:729] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:59.779258 24753 consensus_queue.cc:260] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:59.779349 24753 raft_consensus.cc:3037] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.780303 24753 raft_consensus.cc:2454] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aa9c677a2c84c5082c708ae767eb3a8 in term 2.
I20240502 13:59:59.780710 25952 leader_election.cc:302] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55, 8aa9c677a2c84c5082c708ae767eb3a8; no voters:
I20240502 13:59:59.780913 27457 raft_consensus.cc:2781] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:59.781013 27457 raft_consensus.cc:686] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 LEADER]: Becoming Leader. State: Replica: 8aa9c677a2c84c5082c708ae767eb3a8, State: Running, Role: LEADER
I20240502 13:59:59.781137 27457 consensus_queue.cc:237] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 13:59:59.782794 24607 catalog_manager.cc:5484] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 reported cstate change: term changed from 1 to 2, leader changed from 7b4b1969054146119af3a958ef85de55 (127.22.55.129) to 8aa9c677a2c84c5082c708ae767eb3a8 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 13:59:59.829936 27190 raft_consensus.cc:979] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: : Instructing follower a9e66f5a1b7f4202860a428d70203609 to start an election
I20240502 13:59:59.830039 27190 raft_consensus.cc:1067] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Signalling peer a9e66f5a1b7f4202860a428d70203609 to start an election
I20240502 13:59:59.830489 26411 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cae053fe22fd451e88e8e01eb93b76c5"
dest_uuid: "a9e66f5a1b7f4202860a428d70203609"
from {username='slave'} at 127.22.55.130:37631
I20240502 13:59:59.830618 26411 raft_consensus.cc:482] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:59.830688 26411 raft_consensus.cc:3037] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.831558 26411 raft_consensus.cc:504] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:59.831918 26411 leader_election.cc:288] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [CANDIDATE]: Term 2 election: Requested vote from peers 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 13:59:59.832298 24884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae053fe22fd451e88e8e01eb93b76c5" candidate_uuid: "a9e66f5a1b7f4202860a428d70203609" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
I20240502 13:59:59.832405 24884 raft_consensus.cc:3032] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:59.832454 24884 raft_consensus.cc:729] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 13:59:59.832602 24884 consensus_queue.cc:260] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:59.832722 24884 raft_consensus.cc:3037] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.833434 24884 raft_consensus.cc:2454] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e66f5a1b7f4202860a428d70203609 in term 2.
I20240502 13:59:59.833724 26344 leader_election.cc:302] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6e4fcab25c864479b96f38d291ebedaf, a9e66f5a1b7f4202860a428d70203609; no voters:
I20240502 13:59:59.833878 27496 raft_consensus.cc:2781] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:59.834046 27496 raft_consensus.cc:686] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 2 LEADER]: Becoming Leader. State: Replica: a9e66f5a1b7f4202860a428d70203609, State: Running, Role: LEADER
I20240502 13:59:59.834188 27496 consensus_queue.cc:237] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 13:59:59.835628 24607 catalog_manager.cc:5484] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 reported cstate change: term changed from 1 to 2, leader changed from 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) to a9e66f5a1b7f4202860a428d70203609 (127.22.55.135). New cstate: current_term: 2 leader_uuid: "a9e66f5a1b7f4202860a428d70203609" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 13:59:59.923687 27424 raft_consensus.cc:979] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: : Instructing follower c970e31aa29e4944ab5fb3024984aefd to start an election
I20240502 13:59:59.923802 27424 raft_consensus.cc:1067] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Signalling peer c970e31aa29e4944ab5fb3024984aefd to start an election
I20240502 13:59:59.924248 26280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f943154c90654cd39f602a0fc2b90ceb"
dest_uuid: "c970e31aa29e4944ab5fb3024984aefd"
from {username='slave'} at 127.22.55.129:34637
I20240502 13:59:59.924422 26280 raft_consensus.cc:482] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 13:59:59.924504 26280 raft_consensus.cc:3037] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.925328 26280 raft_consensus.cc:504] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 13:59:59.926122 26280 leader_election.cc:288] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [CANDIDATE]: Term 2 election: Requested vote from peers 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 13:59:59.926474 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f943154c90654cd39f602a0fc2b90ceb" candidate_uuid: "c970e31aa29e4944ab5fb3024984aefd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7b4b1969054146119af3a958ef85de55"
I20240502 13:59:59.926573 24753 raft_consensus.cc:3032] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 13:59:59.926606 24753 raft_consensus.cc:729] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 13:59:59.926717 24753 consensus_queue.cc:260] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 13:59:59.926821 24753 raft_consensus.cc:3037] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:59.927554 24753 raft_consensus.cc:2454] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c970e31aa29e4944ab5fb3024984aefd in term 2.
I20240502 13:59:59.927867 26214 leader_election.cc:302] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55, c970e31aa29e4944ab5fb3024984aefd; no voters:
I20240502 13:59:59.928211 27574 raft_consensus.cc:2781] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:59.928364 27574 raft_consensus.cc:686] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 2 LEADER]: Becoming Leader. State: Replica: c970e31aa29e4944ab5fb3024984aefd, State: Running, Role: LEADER
I20240502 13:59:59.928627 27574 consensus_queue.cc:237] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 13:59:59.930704 24607 catalog_manager.cc:5484] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd reported cstate change: term changed from 1 to 2, leader changed from 7b4b1969054146119af3a958ef85de55 (127.22.55.129) to c970e31aa29e4944ab5fb3024984aefd (127.22.55.134). New cstate: current_term: 2 leader_uuid: "c970e31aa29e4944ab5fb3024984aefd" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 13:59:59.945322 24884 raft_consensus.cc:1261] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Refusing update from remote peer 8aa9c677a2c84c5082c708ae767eb3a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 13:59:59.945772 27456 consensus_queue.cc:1035] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:59.976054 24753 raft_consensus.cc:1261] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Refusing update from remote peer b2a245b6dbf34e03a431a50a7b54ce92: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 13:59:59.976481 27203 consensus_queue.cc:1035] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:00.054461 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.055861 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.056588 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.059896 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:00.060184 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.079885 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.082623 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.117362 26411 consensus_queue.cc:237] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 14:00:00.118058 27496 raft_consensus.cc:2932] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 14:00:00.120167 24602 catalog_manager.cc:5484] T 1ae2eb2957c749b4a34b07944a5c3592 P a9e66f5a1b7f4202860a428d70203609 reported cstate change: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New cstate: current_term: 2 leader_uuid: "a9e66f5a1b7f4202860a428d70203609" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 14:00:00.123073 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae2eb2957c749b4a34b07944a5c3592 with delete_type TABLET_DATA_TOMBSTONED (TS 7b4b1969054146119af3a958ef85de55 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59640
I20240502 14:00:00.123612 27642 tablet_replica.cc:330] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: stopping tablet replica
I20240502 14:00:00.123915 27642 raft_consensus.cc:2229] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:00.124094 27642 raft_consensus.cc:2258] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:00.125046 27642 ts_tablet_manager.cc:1875] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:00.127516 27642 ts_tablet_manager.cc:1888] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:00.127609 27642 log.cc:1196] T 1ae2eb2957c749b4a34b07944a5c3592 P 7b4b1969054146119af3a958ef85de55: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ae2eb2957c749b4a34b07944a5c3592
I20240502 14:00:00.128232 24596 catalog_manager.cc:4830] TS 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649): tablet 1ae2eb2957c749b4a34b07944a5c3592 (table rebalance_test_table_4 [id=2a4b94a9885349e8bb805979c6d23fb0]) successfully deleted
I20240502 14:00:00.176854 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.178033 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.179170 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.179539 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.199515 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.208014 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:00.208119 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.246644 25887 consensus_queue.cc:237] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 14:00:00.247366 27071 raft_consensus.cc:2932] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 14:00:00.249555 24610 catalog_manager.cc:5484] T 94738674b9af470f922b593daec77442 P 51d1006508074f10acd579729457b0c5 reported cstate change: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New cstate: current_term: 2 leader_uuid: "51d1006508074f10acd579729457b0c5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 14:00:00.252420 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet 94738674b9af470f922b593daec77442 with delete_type TABLET_DATA_TOMBSTONED (TS 7b4b1969054146119af3a958ef85de55 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59640
I20240502 14:00:00.252667 27642 tablet_replica.cc:330] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: stopping tablet replica
I20240502 14:00:00.252804 27642 raft_consensus.cc:2229] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:00.252903 27642 raft_consensus.cc:2258] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:00.253532 27642 ts_tablet_manager.cc:1875] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:00.255715 27642 ts_tablet_manager.cc:1888] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:00.255806 27642 log.cc:1196] T 94738674b9af470f922b593daec77442 P 7b4b1969054146119af3a958ef85de55: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/94738674b9af470f922b593daec77442
I20240502 14:00:00.256320 24596 catalog_manager.cc:4830] TS 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649): tablet 94738674b9af470f922b593daec77442 (table rebalance_test_table_2 [id=135ed63340794251874287c06a0d8b85]) successfully deleted
I20240502 14:00:00.284384 24884 raft_consensus.cc:1261] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Refusing update from remote peer a9e66f5a1b7f4202860a428d70203609: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:00:00.284898 27496 consensus_queue.cc:1035] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:00.309677 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.310315 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.310719 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.311889 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.315223 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.326864 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.335702 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:00.370059 25887 consensus_queue.cc:237] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } }
I20240502 14:00:00.370755 27071 raft_consensus.cc:2932] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 14:00:00.373098 24609 catalog_manager.cc:5484] T 27fa46c3e6214af2a67805b7dcfd9301 P 51d1006508074f10acd579729457b0c5 reported cstate change: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New cstate: current_term: 2 leader_uuid: "51d1006508074f10acd579729457b0c5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51d1006508074f10acd579729457b0c5" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 42509 } } }
I20240502 14:00:00.376235 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fa46c3e6214af2a67805b7dcfd9301 with delete_type TABLET_DATA_TOMBSTONED (TS 6e4fcab25c864479b96f38d291ebedaf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48648
I20240502 14:00:00.376731 27753 tablet_replica.cc:330] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: stopping tablet replica
I20240502 14:00:00.377043 27753 raft_consensus.cc:2229] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:00.377207 27753 raft_consensus.cc:2258] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:00.378290 27753 ts_tablet_manager.cc:1875] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:00.380812 27753 ts_tablet_manager.cc:1888] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:00.380898 27753 log.cc:1196] T 27fa46c3e6214af2a67805b7dcfd9301 P 6e4fcab25c864479b96f38d291ebedaf: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/27fa46c3e6214af2a67805b7dcfd9301
I20240502 14:00:00.381531 24595 catalog_manager.cc:4830] TS 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051): tablet 27fa46c3e6214af2a67805b7dcfd9301 (table rebalance_test_table_1 [id=e40cc0c516264dca942c3242c2fdad90]) successfully deleted
I20240502 14:00:00.385428 24753 raft_consensus.cc:1261] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Refusing update from remote peer 8aa9c677a2c84c5082c708ae767eb3a8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:00:00.385824 27457 consensus_queue.cc:1035] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:00.429818 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.433478 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.433540 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.439613 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.441401 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.452258 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.452783 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:00.492076 24753 raft_consensus.cc:1261] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Refusing update from remote peer c970e31aa29e4944ab5fb3024984aefd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:00:00.492697 27792 consensus_queue.cc:1035] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:00.499900 26149 consensus_queue.cc:237] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 14:00:00.500625 27581 raft_consensus.cc:2932] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 14:00:00.503005 24603 catalog_manager.cc:5484] T ff8c1d90d5d8449aac5e3e1e81d54523 P b2a245b6dbf34e03a431a50a7b54ce92 reported cstate change: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New cstate: current_term: 2 leader_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 14:00:00.505704 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8c1d90d5d8449aac5e3e1e81d54523 with delete_type TABLET_DATA_TOMBSTONED (TS 6e4fcab25c864479b96f38d291ebedaf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48648
I20240502 14:00:00.505990 27753 tablet_replica.cc:330] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: stopping tablet replica
I20240502 14:00:00.506145 27753 raft_consensus.cc:2229] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:00.506265 27753 raft_consensus.cc:2258] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:00.506892 27753 ts_tablet_manager.cc:1875] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:00.509063 27753 ts_tablet_manager.cc:1888] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:00.509155 27753 log.cc:1196] T ff8c1d90d5d8449aac5e3e1e81d54523 P 6e4fcab25c864479b96f38d291ebedaf: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/ff8c1d90d5d8449aac5e3e1e81d54523
I20240502 14:00:00.509702 24595 catalog_manager.cc:4830] TS 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051): tablet ff8c1d90d5d8449aac5e3e1e81d54523 (table rebalance_test_table_3 [id=a9389b0cdd3841079087847cdd44fb43]) successfully deleted
I20240502 14:00:00.557947 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.562094 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.564591 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.565119 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.566715 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.567816 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.598675 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:00.632087 26149 consensus_queue.cc:237] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } }
I20240502 14:00:00.632685 27581 raft_consensus.cc:2932] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 14:00:00.634825 24603 catalog_manager.cc:5484] T d465839425704eef962eb345979233f6 P b2a245b6dbf34e03a431a50a7b54ce92 reported cstate change: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New cstate: current_term: 2 leader_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b2a245b6dbf34e03a431a50a7b54ce92" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 38561 } } }
I20240502 14:00:00.637342 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet d465839425704eef962eb345979233f6 with delete_type TABLET_DATA_TOMBSTONED (TS 7b4b1969054146119af3a958ef85de55 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59640
I20240502 14:00:00.637590 27642 tablet_replica.cc:330] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: stopping tablet replica
I20240502 14:00:00.637748 27642 raft_consensus.cc:2229] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:00.637847 27642 raft_consensus.cc:2258] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:00.638497 27642 ts_tablet_manager.cc:1875] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:00.640522 27642 ts_tablet_manager.cc:1888] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:00.640591 27642 log.cc:1196] T d465839425704eef962eb345979233f6 P 7b4b1969054146119af3a958ef85de55: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/d465839425704eef962eb345979233f6
I20240502 14:00:00.641117 24596 catalog_manager.cc:4830] TS 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649): tablet d465839425704eef962eb345979233f6 (table rebalance_test_table_0 [id=ffa197925b7242d892b01e70a6bd7c89]) successfully deleted
I20240502 14:00:00.695024 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.695686 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.696904 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.697165 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.697780 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:00.698979 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.716185 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.756816 26018 consensus_queue.cc:237] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 14:00:00.757422 27457 raft_consensus.cc:2932] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 14:00:00.759979 24607 catalog_manager.cc:5484] T 94e5deba02294d4e85fd4bad586a78d5 P 8aa9c677a2c84c5082c708ae767eb3a8 reported cstate change: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New cstate: current_term: 2 leader_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 14:00:00.762367 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e5deba02294d4e85fd4bad586a78d5 with delete_type TABLET_DATA_TOMBSTONED (TS 6e4fcab25c864479b96f38d291ebedaf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48648
I20240502 14:00:00.762578 27753 tablet_replica.cc:330] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: stopping tablet replica
I20240502 14:00:00.762697 27753 raft_consensus.cc:2229] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:00.762809 27753 raft_consensus.cc:2258] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:00.763530 27753 ts_tablet_manager.cc:1875] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:00.765805 27753 ts_tablet_manager.cc:1888] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:00.765884 27753 log.cc:1196] T 94e5deba02294d4e85fd4bad586a78d5 P 6e4fcab25c864479b96f38d291ebedaf: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/94e5deba02294d4e85fd4bad586a78d5
I20240502 14:00:00.766399 24595 catalog_manager.cc:4830] TS 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051): tablet 94e5deba02294d4e85fd4bad586a78d5 (table rebalance_test_table_3 [id=a9389b0cdd3841079087847cdd44fb43]) successfully deleted
I20240502 14:00:00.816694 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.819769 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.823580 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:00.824803 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.835934 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.847273 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.853323 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.888337 26018 consensus_queue.cc:237] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } }
I20240502 14:00:00.888927 27457 raft_consensus.cc:2932] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 14:00:00.891076 24603 catalog_manager.cc:5484] T b81678740ddc4fce9da067dc3d8dff8d P 8aa9c677a2c84c5082c708ae767eb3a8 reported cstate change: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New cstate: current_term: 2 leader_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8aa9c677a2c84c5082c708ae767eb3a8" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 39445 } } }
I20240502 14:00:00.893532 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet b81678740ddc4fce9da067dc3d8dff8d with delete_type TABLET_DATA_TOMBSTONED (TS 7b4b1969054146119af3a958ef85de55 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59640
I20240502 14:00:00.893785 27642 tablet_replica.cc:330] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: stopping tablet replica
I20240502 14:00:00.893939 27642 raft_consensus.cc:2229] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:00.894052 27642 raft_consensus.cc:2258] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:00.894750 27642 ts_tablet_manager.cc:1875] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:00.896842 27642 ts_tablet_manager.cc:1888] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:00.896916 27642 log.cc:1196] T b81678740ddc4fce9da067dc3d8dff8d P 7b4b1969054146119af3a958ef85de55: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/b81678740ddc4fce9da067dc3d8dff8d
I20240502 14:00:00.897409 24596 catalog_manager.cc:4830] TS 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649): tablet b81678740ddc4fce9da067dc3d8dff8d (table rebalance_test_table_0 [id=ffa197925b7242d892b01e70a6bd7c89]) successfully deleted
I20240502 14:00:00.947966 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.950871 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.951221 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.951382 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:00.959035 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:00.959779 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:00.979540 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:01.015630 26411 consensus_queue.cc:237] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } }
I20240502 14:00:01.016230 27572 raft_consensus.cc:2932] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 14:00:01.018612 24602 catalog_manager.cc:5484] T cae053fe22fd451e88e8e01eb93b76c5 P a9e66f5a1b7f4202860a428d70203609 reported cstate change: config changed from index 4 to 6, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New cstate: current_term: 2 leader_uuid: "a9e66f5a1b7f4202860a428d70203609" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9e66f5a1b7f4202860a428d70203609" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 41681 } } }
I20240502 14:00:01.021167 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet cae053fe22fd451e88e8e01eb93b76c5 with delete_type TABLET_DATA_TOMBSTONED (TS 6e4fcab25c864479b96f38d291ebedaf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48648
I20240502 14:00:01.021404 27753 tablet_replica.cc:330] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: stopping tablet replica
I20240502 14:00:01.021557 27753 raft_consensus.cc:2229] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:01.021670 27753 raft_consensus.cc:2258] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:01.022334 27753 ts_tablet_manager.cc:1875] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:01.024612 27753 ts_tablet_manager.cc:1888] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:01.024683 27753 log.cc:1196] T cae053fe22fd451e88e8e01eb93b76c5 P 6e4fcab25c864479b96f38d291ebedaf: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/cae053fe22fd451e88e8e01eb93b76c5
I20240502 14:00:01.025172 24595 catalog_manager.cc:4830] TS 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051): tablet cae053fe22fd451e88e8e01eb93b76c5 (table rebalance_test_table_1 [id=e40cc0c516264dca942c3242c2fdad90]) successfully deleted
I20240502 14:00:01.076464 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:01.077795 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:01.078361 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:01.078522 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:01.083421 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:01.094496 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:01.107301 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:01.147172 26280 consensus_queue.cc:237] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 14:00:01.147742 27792 raft_consensus.cc:2932] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:01.150449 24608 catalog_manager.cc:5484] T f943154c90654cd39f602a0fc2b90ceb P c970e31aa29e4944ab5fb3024984aefd reported cstate change: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New cstate: current_term: 2 leader_uuid: "c970e31aa29e4944ab5fb3024984aefd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:01.152915 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet f943154c90654cd39f602a0fc2b90ceb with delete_type TABLET_DATA_TOMBSTONED (TS 7b4b1969054146119af3a958ef85de55 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59640
I20240502 14:00:01.153146 27642 tablet_replica.cc:330] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: stopping tablet replica
I20240502 14:00:01.153263 27642 raft_consensus.cc:2229] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:01.153376 27642 raft_consensus.cc:2258] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:01.154035 27642 ts_tablet_manager.cc:1875] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:01.156002 27642 ts_tablet_manager.cc:1888] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:01.156095 27642 log.cc:1196] T f943154c90654cd39f602a0fc2b90ceb P 7b4b1969054146119af3a958ef85de55: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/f943154c90654cd39f602a0fc2b90ceb
I20240502 14:00:01.156589 24596 catalog_manager.cc:4830] TS 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649): tablet f943154c90654cd39f602a0fc2b90ceb (table rebalance_test_table_4 [id=2a4b94a9885349e8bb805979c6d23fb0]) successfully deleted
I20240502 14:00:01.413672 24753 consensus_queue.cc:237] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
W20240502 14:00:01.415005 24688 consensus_peers.cc:485] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 -> Peer c970e31aa29e4944ab5fb3024984aefd (127.22.55.134:41009): Couldn't send request to peer c970e31aa29e4944ab5fb3024984aefd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ca9e7a8cb7644f48a634b8ef86cad9f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:01.876672 28132 ts_tablet_manager.cc:902] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: Initiating tablet copy from peer 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 14:00:01.877197 28132 tablet_copy_client.cc:308] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:36649
I20240502 14:00:01.877753 24773 tablet_copy_service.cc:140] P 7b4b1969054146119af3a958ef85de55: Received BeginTabletCopySession request for tablet 6ca9e7a8cb7644f48a634b8ef86cad9f from peer c970e31aa29e4944ab5fb3024984aefd ({username='slave'} at 127.22.55.134:56997)
I20240502 14:00:01.877868 24773 tablet_copy_service.cc:161] P 7b4b1969054146119af3a958ef85de55: Beginning new tablet copy session on tablet 6ca9e7a8cb7644f48a634b8ef86cad9f from peer c970e31aa29e4944ab5fb3024984aefd at {username='slave'} at 127.22.55.134:56997: session id = c970e31aa29e4944ab5fb3024984aefd-6ca9e7a8cb7644f48a634b8ef86cad9f
I20240502 14:00:01.878705 24773 tablet_copy_source_session.cc:205] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:01.879614 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca9e7a8cb7644f48a634b8ef86cad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.881341 28132 tablet_copy_client.cc:783] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:01.881486 28132 tablet_copy_client.cc:647] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:01.882556 28132 tablet_copy_client.cc:523] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:01.883555 28132 tablet_bootstrap.cc:492] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: Bootstrap starting.
I20240502 14:00:01.901227 28132 tablet_bootstrap.cc:492] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:01.901535 28132 tablet_bootstrap.cc:492] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: Bootstrap complete.
I20240502 14:00:01.901702 28132 ts_tablet_manager.cc:1367] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240502 14:00:01.902221 28132 raft_consensus.cc:348] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 14:00:01.902434 28132 raft_consensus.cc:729] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c970e31aa29e4944ab5fb3024984aefd, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.902537 28132 consensus_queue.cc:260] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 14:00:01.902921 28132 ts_tablet_manager.cc:1398] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:00:01.903512 24773 tablet_copy_service.cc:342] P 7b4b1969054146119af3a958ef85de55: Request end of tablet copy session c970e31aa29e4944ab5fb3024984aefd-6ca9e7a8cb7644f48a634b8ef86cad9f received from {username='slave'} at 127.22.55.134:56997
I20240502 14:00:01.903601 24773 tablet_copy_service.cc:433] P 7b4b1969054146119af3a958ef85de55: ending tablet copy session c970e31aa29e4944ab5fb3024984aefd-6ca9e7a8cb7644f48a634b8ef86cad9f on tablet 6ca9e7a8cb7644f48a634b8ef86cad9f with peer c970e31aa29e4944ab5fb3024984aefd
I20240502 14:00:02.436547 26280 raft_consensus.cc:1203] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:02.437551 28141 raft_consensus.cc:2932] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c970e31aa29e4944ab5fb3024984aefd (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:02.438012 26280 raft_consensus.cc:2932] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c970e31aa29e4944ab5fb3024984aefd (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:02.439225 28141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:48426 (request call id 0) took 1025 ms. Trace:
I20240502 14:00:02.439342 28141 rpcz_store.cc:276] 0502 14:00:01.413335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:00:01.413399 (+ 64us) service_pool.cc:225] Handling call
0502 14:00:01.414049 (+ 650us) log.cc:842] Serialized 260 byte log entry
0502 14:00:02.439205 (+1025156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"thread_start_us":104,"threads_started":2,"wal-append.queue_time_us":130,"wal-append.run_cpu_time_us":340,"wal-append.run_wall_time_us":1000500}
I20240502 14:00:02.440728 24608 catalog_manager.cc:5484] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd reported cstate change: config changed from index -1 to 4, VOTER c970e31aa29e4944ab5fb3024984aefd (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:02.501345 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.501473 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:02.502909 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.503836 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:02.504623 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.515375 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.527716 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:02.551059 24753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ca9e7a8cb7644f48a634b8ef86cad9f"
dest_uuid: "7b4b1969054146119af3a958ef85de55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40242
I20240502 14:00:02.551203 24753 raft_consensus.cc:595] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:02.601918 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.604100 24732 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:02.606282 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.607750 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:02.615222 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.615818 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.625149 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:02.840628 28139 raft_consensus.cc:979] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: : Instructing follower c970e31aa29e4944ab5fb3024984aefd to start an election
I20240502 14:00:02.840766 28139 raft_consensus.cc:1067] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Signalling peer c970e31aa29e4944ab5fb3024984aefd to start an election
I20240502 14:00:02.841226 26280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ca9e7a8cb7644f48a634b8ef86cad9f"
dest_uuid: "c970e31aa29e4944ab5fb3024984aefd"
from {username='slave'} at 127.22.55.129:34637
I20240502 14:00:02.841367 26280 raft_consensus.cc:482] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:02.841434 26280 raft_consensus.cc:3037] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:02.842238 26280 raft_consensus.cc:504] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 14:00:02.842541 26280 leader_election.cc:288] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [CANDIDATE]: Term 2 election: Requested vote from peers 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 14:00:02.842929 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca9e7a8cb7644f48a634b8ef86cad9f" candidate_uuid: "c970e31aa29e4944ab5fb3024984aefd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7b4b1969054146119af3a958ef85de55"
I20240502 14:00:02.843060 24753 raft_consensus.cc:3032] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:02.843096 24753 raft_consensus.cc:729] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 14:00:02.843215 24753 consensus_queue.cc:260] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 14:00:02.843327 24753 raft_consensus.cc:3037] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:02.844022 24753 raft_consensus.cc:2454] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c970e31aa29e4944ab5fb3024984aefd in term 2.
I20240502 14:00:02.844296 26214 leader_election.cc:302] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b4b1969054146119af3a958ef85de55, c970e31aa29e4944ab5fb3024984aefd; no voters:
I20240502 14:00:02.844468 28140 raft_consensus.cc:2781] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:02.844617 28140 raft_consensus.cc:686] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 2 LEADER]: Becoming Leader. State: Replica: c970e31aa29e4944ab5fb3024984aefd, State: Running, Role: LEADER
I20240502 14:00:02.844736 28140 consensus_queue.cc:237] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 14:00:02.846072 24602 catalog_manager.cc:5484] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd reported cstate change: term changed from 1 to 2, leader changed from 7b4b1969054146119af3a958ef85de55 (127.22.55.129) to c970e31aa29e4944ab5fb3024984aefd (127.22.55.134). New cstate: current_term: 2 leader_uuid: "c970e31aa29e4944ab5fb3024984aefd" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:02.909039 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.914471 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:02.917068 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.920192 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:02.922258 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.927604 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:02.928467 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.218588 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.221957 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.227684 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.228636 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.229779 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.235294 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.244422 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.377784 24753 raft_consensus.cc:1261] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Refusing update from remote peer c970e31aa29e4944ab5fb3024984aefd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:00:03.378252 28305 consensus_queue.cc:1035] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:03.530315 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.530735 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.532054 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.543521 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.544868 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.551832 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.557767 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.596808 26280 consensus_queue.cc:237] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } }
I20240502 14:00:03.597419 28307 raft_consensus.cc:2932] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:03.599606 24603 catalog_manager.cc:5484] T 6ca9e7a8cb7644f48a634b8ef86cad9f P c970e31aa29e4944ab5fb3024984aefd reported cstate change: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New cstate: current_term: 2 leader_uuid: "c970e31aa29e4944ab5fb3024984aefd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c970e31aa29e4944ab5fb3024984aefd" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 41009 } } }
I20240502 14:00:03.602164 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca9e7a8cb7644f48a634b8ef86cad9f with delete_type TABLET_DATA_TOMBSTONED (TS 7b4b1969054146119af3a958ef85de55 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59640
I20240502 14:00:03.602531 28365 tablet_replica.cc:330] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: stopping tablet replica
I20240502 14:00:03.602664 28365 raft_consensus.cc:2229] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:03.602778 28365 raft_consensus.cc:2258] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:03.603367 28365 ts_tablet_manager.cc:1875] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:03.605132 28365 ts_tablet_manager.cc:1888] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:03.605202 28365 log.cc:1196] T 6ca9e7a8cb7644f48a634b8ef86cad9f P 7b4b1969054146119af3a958ef85de55: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/6ca9e7a8cb7644f48a634b8ef86cad9f
I20240502 14:00:03.605729 24596 catalog_manager.cc:4830] TS 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649): tablet 6ca9e7a8cb7644f48a634b8ef86cad9f (table rebalance_test_table_2 [id=135ed63340794251874287c06a0d8b85]) successfully deleted
I20240502 14:00:03.861012 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.867944 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.873878 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.875316 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.876659 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.894865 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.898710 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.973630 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.979532 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.979784 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.979936 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.981202 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:03.981491 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.006723 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.243959 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:04.251041 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:04.254043 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.259295 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.259980 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.270900 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.279968 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.406988 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:04.408480 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.408666 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:04.411422 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.416277 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.419936 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.425302 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.526422 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.526515 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:04.528724 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.531450 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:04.536590 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.543745 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:04.547945 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
84497bd32f594599b786a42799ec8d7a | 127.22.55.190:38111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.55.148:41981 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
51d1006508074f10acd579729457b0c5 | 127.22.55.131:42509 | HEALTHY | <none> | 2 | 0
6e4fcab25c864479b96f38d291ebedaf | 127.22.55.130:38051 | HEALTHY | <none> | 3 | 0
7b4b1969054146119af3a958ef85de55 | 127.22.55.129:36649 | HEALTHY | <none> | 3 | 0
8aa9c677a2c84c5082c708ae767eb3a8 | 127.22.55.132:39445 | HEALTHY | <none> | 2 | 0
a9e66f5a1b7f4202860a428d70203609 | 127.22.55.135:41681 | HEALTHY | <none> | 2 | 0
b2a245b6dbf34e03a431a50a7b54ce92 | 127.22.55.133:38561 | HEALTHY | <none> | 2 | 0
c970e31aa29e4944ab5fb3024984aefd | 127.22.55.134:41009 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.22.55.129 | experimental | 127.22.55.129:36649
local_ip_for_outbound_sockets | 127.22.55.130 | experimental | 127.22.55.130:38051
local_ip_for_outbound_sockets | 127.22.55.131 | experimental | 127.22.55.131:42509
local_ip_for_outbound_sockets | 127.22.55.132 | experimental | 127.22.55.132:39445
local_ip_for_outbound_sockets | 127.22.55.133 | experimental | 127.22.55.133:38561
local_ip_for_outbound_sockets | 127.22.55.134 | experimental | 127.22.55.134:41009
local_ip_for_outbound_sockets | 127.22.55.135 | experimental | 127.22.55.135:41681
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.55.129:36649
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.55.130:38051
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.55.131:42509
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.55.132:39445
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.55.133:38561
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.55.134:41009
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.55.135:41681
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.55.148:41981 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:00:04.578868 22750 log_verifier.cc:126] Checking tablet 00c8e8a7c7a6493088d08577906bf21c
I20240502 14:00:04.590111 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 00c8e8a7c7a6493088d08577906bf21c
I20240502 14:00:04.590211 22750 log_verifier.cc:126] Checking tablet 07dbaad8ae36426b9eff6ea62db21dc7
I20240502 14:00:04.598387 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 07dbaad8ae36426b9eff6ea62db21dc7
I20240502 14:00:04.598462 22750 log_verifier.cc:126] Checking tablet 0cad80f3ea304711b302663a1f420aba
I20240502 14:00:04.606681 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0cad80f3ea304711b302663a1f420aba
I20240502 14:00:04.606747 22750 log_verifier.cc:126] Checking tablet 1ae2eb2957c749b4a34b07944a5c3592
I20240502 14:00:04.615144 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1ae2eb2957c749b4a34b07944a5c3592
I20240502 14:00:04.615226 22750 log_verifier.cc:126] Checking tablet 27fa46c3e6214af2a67805b7dcfd9301
I20240502 14:00:04.623840 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 27fa46c3e6214af2a67805b7dcfd9301
I20240502 14:00:04.623906 22750 log_verifier.cc:126] Checking tablet 4fe45162fdb94d61961fea1cb7cc6af8
I20240502 14:00:04.632213 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4fe45162fdb94d61961fea1cb7cc6af8
I20240502 14:00:04.632279 22750 log_verifier.cc:126] Checking tablet 6ca9e7a8cb7644f48a634b8ef86cad9f
I20240502 14:00:04.640511 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6ca9e7a8cb7644f48a634b8ef86cad9f
I20240502 14:00:04.640576 22750 log_verifier.cc:126] Checking tablet 75834b9995394a2d95a56acc95eea043
I20240502 14:00:04.648622 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 75834b9995394a2d95a56acc95eea043
I20240502 14:00:04.648685 22750 log_verifier.cc:126] Checking tablet 94738674b9af470f922b593daec77442
I20240502 14:00:04.656957 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 94738674b9af470f922b593daec77442
I20240502 14:00:04.657019 22750 log_verifier.cc:126] Checking tablet 94e5deba02294d4e85fd4bad586a78d5
I20240502 14:00:04.665266 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 94e5deba02294d4e85fd4bad586a78d5
I20240502 14:00:04.665328 22750 log_verifier.cc:126] Checking tablet ade0bed961d448449904f531c57ad940
I20240502 14:00:04.672603 22750 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ade0bed961d448449904f531c57ad940
I20240502 14:00:04.672674 22750 log_verifier.cc:126] Checking tablet b81678740ddc4fce9da067dc3d8dff8d
I20240502 14:00:04.682803 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b81678740ddc4fce9da067dc3d8dff8d
I20240502 14:00:04.682932 22750 log_verifier.cc:126] Checking tablet cae053fe22fd451e88e8e01eb93b76c5
I20240502 14:00:04.691577 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet cae053fe22fd451e88e8e01eb93b76c5
I20240502 14:00:04.691645 22750 log_verifier.cc:126] Checking tablet d465839425704eef962eb345979233f6
I20240502 14:00:04.700073 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d465839425704eef962eb345979233f6
I20240502 14:00:04.700143 22750 log_verifier.cc:126] Checking tablet f943154c90654cd39f602a0fc2b90ceb
I20240502 14:00:04.708335 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f943154c90654cd39f602a0fc2b90ceb
I20240502 14:00:04.708402 22750 log_verifier.cc:126] Checking tablet ff8c1d90d5d8449aac5e3e1e81d54523
I20240502 14:00:04.716641 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ff8c1d90d5d8449aac5e3e1e81d54523
I20240502 14:00:04.718701 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.136:0
--local_ip_for_outbound_sockets=127.22.55.136
--webserver_interface=127.22.55.136
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--builtin_ntp_servers=127.22.55.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:04.826805 28541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:04.827080 28541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:04.827133 28541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:04.830219 28541 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:04.830278 28541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:04.830367 28541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.136
I20240502 14:00:04.834041 28541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.55.136
--webserver_port=0
--tserver_master_addrs=127.22.55.190:38111
--never_fsync=true
--heap_profile_path=/tmp/kudu.28541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.136
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:04.835057 28541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:04.836221 28541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:00:04.845661 28541 server_base.cc:1034] running on GCE node
W20240502 14:00:04.846614 28546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:04.847391 28549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:04.848292 28547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:04.848729 28541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:04.849328 28541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:04.850478 28541 hybrid_clock.cc:648] HybridClock initialized: now 1714658404850466 us; error 33 us; skew 500 ppm
I20240502 14:00:04.852414 28541 webserver.cc:460] Webserver started at http://127.22.55.136:39775/ using document root <none> and password file <none>
I20240502 14:00:04.853268 28541 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:04.853356 28541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:04.853622 28541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:04.855535 28541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "34ee00b6f4554b39ab5e5d1a0252fae1"
format_stamp: "Formatted at 2024-05-02 14:00:04 on dist-test-slave-sdf5"
I20240502 14:00:04.855994 28541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "34ee00b6f4554b39ab5e5d1a0252fae1"
format_stamp: "Formatted at 2024-05-02 14:00:04 on dist-test-slave-sdf5"
I20240502 14:00:04.859354 28541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:00:04.861546 28555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:04.862540 28541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:04.862638 28541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "34ee00b6f4554b39ab5e5d1a0252fae1"
format_stamp: "Formatted at 2024-05-02 14:00:04 on dist-test-slave-sdf5"
I20240502 14:00:04.862728 28541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:04.878074 28541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:04.878726 28541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:04.878945 28541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:04.879563 28541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:04.880453 28541 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:04.880499 28541 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:04.880546 28541 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:04.880575 28541 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:04.889123 28541 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.136:45809
I20240502 14:00:04.889145 28668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.136:45809 every 8 connection(s)
I20240502 14:00:04.890101 28541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:00:04.894681 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 28541
I20240502 14:00:04.894865 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:00:04.899608 28669 heartbeater.cc:347] Connected to a master server at 127.22.55.190:38111
I20240502 14:00:04.899889 28669 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:04.900437 28669 heartbeater.cc:511] Master 127.22.55.190:38111 requested a full tablet report, sending...
I20240502 14:00:04.901487 24607 ts_manager.cc:194] Registered new tserver with Master: 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136:45809)
I20240502 14:00:04.902052 24607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.136:35187
I20240502 14:00:05.091279 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:05.093009 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:05.093184 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.099706 28603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:05.103030 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.103272 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.121649 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.122265 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.224349 28603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:05.230068 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.232165 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.232455 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.235060 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.238566 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:05.239534 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:05.245472 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:05.301836 24753 consensus_queue.cc:237] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
W20240502 14:00:05.306784 24686 consensus_peers.cc:485] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 -> Peer 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136:45809): Couldn't send request to peer 34ee00b6f4554b39ab5e5d1a0252fae1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75834b9995394a2d95a56acc95eea043. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:05.754092 28774 ts_tablet_manager.cc:902] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: Initiating tablet copy from peer 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 14:00:05.754909 28774 tablet_copy_client.cc:308] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:36649
I20240502 14:00:05.758785 24773 tablet_copy_service.cc:140] P 7b4b1969054146119af3a958ef85de55: Received BeginTabletCopySession request for tablet 75834b9995394a2d95a56acc95eea043 from peer 34ee00b6f4554b39ab5e5d1a0252fae1 ({username='slave'} at 127.22.55.136:51239)
I20240502 14:00:05.758934 24773 tablet_copy_service.cc:161] P 7b4b1969054146119af3a958ef85de55: Beginning new tablet copy session on tablet 75834b9995394a2d95a56acc95eea043 from peer 34ee00b6f4554b39ab5e5d1a0252fae1 at {username='slave'} at 127.22.55.136:51239: session id = 34ee00b6f4554b39ab5e5d1a0252fae1-75834b9995394a2d95a56acc95eea043
I20240502 14:00:05.759748 24773 tablet_copy_source_session.cc:205] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:05.762951 28774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75834b9995394a2d95a56acc95eea043. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.768170 28774 tablet_copy_client.cc:783] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:05.768375 28774 tablet_copy_client.cc:647] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:05.770069 28774 tablet_copy_client.cc:523] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:05.771742 28774 tablet_bootstrap.cc:492] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: Bootstrap starting.
I20240502 14:00:05.777724 28774 log.cc:824] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:05.800380 28774 tablet_bootstrap.cc:492] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:05.800788 28774 tablet_bootstrap.cc:492] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: Bootstrap complete.
I20240502 14:00:05.801103 28774 ts_tablet_manager.cc:1367] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20240502 14:00:05.803140 28774 raft_consensus.cc:348] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } }
I20240502 14:00:05.803699 28774 raft_consensus.cc:729] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ee00b6f4554b39ab5e5d1a0252fae1, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.803985 28774 consensus_queue.cc:260] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:05.804701 28669 heartbeater.cc:503] Master 127.22.55.190:38111 was elected leader, sending a full tablet report...
I20240502 14:00:05.804903 28774 ts_tablet_manager.cc:1398] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 14:00:05.805624 24773 tablet_copy_service.cc:342] P 7b4b1969054146119af3a958ef85de55: Request end of tablet copy session 34ee00b6f4554b39ab5e5d1a0252fae1-75834b9995394a2d95a56acc95eea043 received from {username='slave'} at 127.22.55.136:51239
I20240502 14:00:05.805707 24773 tablet_copy_service.cc:433] P 7b4b1969054146119af3a958ef85de55: ending tablet copy session 34ee00b6f4554b39ab5e5d1a0252fae1-75834b9995394a2d95a56acc95eea043 on tablet 75834b9995394a2d95a56acc95eea043 with peer 34ee00b6f4554b39ab5e5d1a0252fae1
I20240502 14:00:06.329886 28623 raft_consensus.cc:1203] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:06.331187 28784 raft_consensus.cc:2932] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:06.331748 28623 raft_consensus.cc:2932] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:06.332698 28784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40342 (request call id 0) took 1031 ms. Trace:
I20240502 14:00:06.332819 28784 rpcz_store.cc:276] 0502 14:00:05.301474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:00:05.301536 (+ 62us) service_pool.cc:225] Handling call
0502 14:00:05.302342 (+ 806us) log.cc:842] Serialized 260 byte log entry
0502 14:00:06.332682 (+1030340us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":57,"raft.queue_time_us":115,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":111,"threads_started":2,"wal-append.queue_time_us":155,"wal-append.run_cpu_time_us":320,"wal-append.run_wall_time_us":1000386}
I20240502 14:00:06.334043 24602 catalog_manager.cc:5484] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 reported cstate change: config changed from index -1 to 4, VOTER 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "7b4b1969054146119af3a958ef85de55" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:06.351789 24884 consensus_queue.cc:237] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
W20240502 14:00:06.355926 24817 consensus_peers.cc:485] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf -> Peer 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136:45809): Couldn't send request to peer 34ee00b6f4554b39ab5e5d1a0252fae1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade0bed961d448449904f531c57ad940. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:06.796515 28812 ts_tablet_manager.cc:902] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: Initiating tablet copy from peer 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 14:00:06.797041 28812 tablet_copy_client.cc:308] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:38051
I20240502 14:00:06.800776 24904 tablet_copy_service.cc:140] P 6e4fcab25c864479b96f38d291ebedaf: Received BeginTabletCopySession request for tablet ade0bed961d448449904f531c57ad940 from peer 34ee00b6f4554b39ab5e5d1a0252fae1 ({username='slave'} at 127.22.55.136:50065)
I20240502 14:00:06.800879 24904 tablet_copy_service.cc:161] P 6e4fcab25c864479b96f38d291ebedaf: Beginning new tablet copy session on tablet ade0bed961d448449904f531c57ad940 from peer 34ee00b6f4554b39ab5e5d1a0252fae1 at {username='slave'} at 127.22.55.136:50065: session id = 34ee00b6f4554b39ab5e5d1a0252fae1-ade0bed961d448449904f531c57ad940
I20240502 14:00:06.801616 24904 tablet_copy_source_session.cc:205] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:06.802412 28812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade0bed961d448449904f531c57ad940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:06.804148 28812 tablet_copy_client.cc:783] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:06.804284 28812 tablet_copy_client.cc:647] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:06.805390 28812 tablet_copy_client.cc:523] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:06.806298 28812 tablet_bootstrap.cc:492] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: Bootstrap starting.
I20240502 14:00:06.808928 28812 tablet_bootstrap.cc:492] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:06.809146 28812 tablet_bootstrap.cc:492] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: Bootstrap complete.
I20240502 14:00:06.809284 28812 ts_tablet_manager.cc:1367] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:00:06.809890 28812 raft_consensus.cc:348] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } }
I20240502 14:00:06.810079 28812 raft_consensus.cc:729] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ee00b6f4554b39ab5e5d1a0252fae1, State: Initialized, Role: FOLLOWER
I20240502 14:00:06.810187 28812 consensus_queue.cc:260] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:06.810421 28812 ts_tablet_manager.cc:1398] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:06.810986 24904 tablet_copy_service.cc:342] P 6e4fcab25c864479b96f38d291ebedaf: Request end of tablet copy session 34ee00b6f4554b39ab5e5d1a0252fae1-ade0bed961d448449904f531c57ad940 received from {username='slave'} at 127.22.55.136:50065
I20240502 14:00:06.811076 24904 tablet_copy_service.cc:433] P 6e4fcab25c864479b96f38d291ebedaf: ending tablet copy session 34ee00b6f4554b39ab5e5d1a0252fae1-ade0bed961d448449904f531c57ad940 on tablet ade0bed961d448449904f531c57ad940 with peer 34ee00b6f4554b39ab5e5d1a0252fae1
I20240502 14:00:07.379120 28623 raft_consensus.cc:1203] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:07.380264 28821 raft_consensus.cc:2932] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:07.380553 28623 raft_consensus.cc:2932] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:07.381915 28821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33356 (request call id 0) took 1030 ms. Trace:
I20240502 14:00:07.382046 28821 rpcz_store.cc:276] 0502 14:00:06.351486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:00:06.351538 (+ 52us) service_pool.cc:225] Handling call
0502 14:00:06.352169 (+ 631us) log.cc:842] Serialized 260 byte log entry
0502 14:00:07.381897 (+1029728us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":37,"raft.queue_time_us":105,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"thread_start_us":106,"threads_started":2,"wal-append.queue_time_us":137,"wal-append.run_cpu_time_us":420,"wal-append.run_wall_time_us":1000463}
I20240502 14:00:07.383476 24602 catalog_manager.cc:5484] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf reported cstate change: config changed from index -1 to 2, VOTER 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "6e4fcab25c864479b96f38d291ebedaf" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:07.441591 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.442461 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.442698 28603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.443759 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.448576 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.454527 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.455355 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.483985 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.506887 24884 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ade0bed961d448449904f531c57ad940"
dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33394
I20240502 14:00:07.507033 24884 raft_consensus.cc:595] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:07.556396 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.561537 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.562911 28603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.567409 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.567992 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.575534 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.576839 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.585269 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.671276 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.676419 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.676916 28603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.684881 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.686926 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.687186 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.689875 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.710307 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.734110 24753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75834b9995394a2d95a56acc95eea043"
dest_uuid: "7b4b1969054146119af3a958ef85de55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40390
I20240502 14:00:07.734256 24753 raft_consensus.cc:595] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:07.783885 28603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.791056 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.792958 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.797598 24863 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.800182 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.806228 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:07.814693 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.831064 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:07.849514 28821 raft_consensus.cc:979] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: : Instructing follower 34ee00b6f4554b39ab5e5d1a0252fae1 to start an election
I20240502 14:00:07.849639 28821 raft_consensus.cc:1067] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Signalling peer 34ee00b6f4554b39ab5e5d1a0252fae1 to start an election
I20240502 14:00:07.851063 28623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ade0bed961d448449904f531c57ad940"
dest_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1"
from {username='slave'} at 127.22.55.130:51729
I20240502 14:00:07.851316 28623 raft_consensus.cc:482] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:07.851418 28623 raft_consensus.cc:3037] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:07.852543 28623 raft_consensus.cc:504] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:07.853426 28623 leader_election.cc:288] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [CANDIDATE]: Term 2 election: Requested vote from peers 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051)
I20240502 14:00:07.853889 24884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade0bed961d448449904f531c57ad940" candidate_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6e4fcab25c864479b96f38d291ebedaf"
I20240502 14:00:07.854017 24884 raft_consensus.cc:3032] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:07.854064 24884 raft_consensus.cc:729] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e4fcab25c864479b96f38d291ebedaf, State: Running, Role: LEADER
I20240502 14:00:07.854214 24884 consensus_queue.cc:260] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:07.854341 24884 raft_consensus.cc:3037] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:07.855329 24884 raft_consensus.cc:2454] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ee00b6f4554b39ab5e5d1a0252fae1 in term 2.
I20240502 14:00:07.856127 28556 leader_election.cc:302] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34ee00b6f4554b39ab5e5d1a0252fae1, 6e4fcab25c864479b96f38d291ebedaf; no voters:
I20240502 14:00:07.856432 28820 raft_consensus.cc:2781] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:07.856640 28820 raft_consensus.cc:686] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 LEADER]: Becoming Leader. State: Replica: 34ee00b6f4554b39ab5e5d1a0252fae1, State: Running, Role: LEADER
I20240502 14:00:07.856930 28820 consensus_queue.cc:237] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:07.858956 24610 catalog_manager.cc:5484] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 reported cstate change: term changed from 1 to 2, leader changed from 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) to 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:08.114821 28603 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:08.114934 24732 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:08.116911 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.122973 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.122969 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.123189 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.140777 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.152365 28784 raft_consensus.cc:979] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: : Instructing follower 34ee00b6f4554b39ab5e5d1a0252fae1 to start an election
I20240502 14:00:08.152464 28784 raft_consensus.cc:1067] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Signalling peer 34ee00b6f4554b39ab5e5d1a0252fae1 to start an election
I20240502 14:00:08.152866 28623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75834b9995394a2d95a56acc95eea043"
dest_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1"
from {username='slave'} at 127.22.55.129:57455
I20240502 14:00:08.153026 28623 raft_consensus.cc:482] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:08.153080 28623 raft_consensus.cc:3037] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:08.154134 28623 raft_consensus.cc:504] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:08.154507 28623 leader_election.cc:288] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [CANDIDATE]: Term 2 election: Requested vote from peers 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649)
I20240502 14:00:08.154958 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75834b9995394a2d95a56acc95eea043" candidate_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7b4b1969054146119af3a958ef85de55"
I20240502 14:00:08.155083 24753 raft_consensus.cc:3032] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:08.155133 24753 raft_consensus.cc:729] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b4b1969054146119af3a958ef85de55, State: Running, Role: LEADER
I20240502 14:00:08.155272 24753 consensus_queue.cc:260] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:08.155405 24753 raft_consensus.cc:3037] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:08.156410 24753 raft_consensus.cc:2454] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ee00b6f4554b39ab5e5d1a0252fae1 in term 2.
I20240502 14:00:08.156764 28557 leader_election.cc:302] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 34ee00b6f4554b39ab5e5d1a0252fae1, 7b4b1969054146119af3a958ef85de55; no voters:
I20240502 14:00:08.156982 28820 raft_consensus.cc:2781] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:08.157047 28820 raft_consensus.cc:686] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 LEADER]: Becoming Leader. State: Replica: 34ee00b6f4554b39ab5e5d1a0252fae1, State: Running, Role: LEADER
I20240502 14:00:08.157156 28820 consensus_queue.cc:237] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:08.158160 24863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.158808 24610 catalog_manager.cc:5484] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 reported cstate change: term changed from 1 to 2, leader changed from 7b4b1969054146119af3a958ef85de55 (127.22.55.129) to 34ee00b6f4554b39ab5e5d1a0252fae1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 } } peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:08.231266 24732 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.234930 28603 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.236048 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.236598 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.244252 24863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.247776 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.247874 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.257402 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.473752 24884 raft_consensus.cc:1261] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Refusing update from remote peer 34ee00b6f4554b39ab5e5d1a0252fae1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:00:08.474328 28989 consensus_queue.cc:1035] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4fcab25c864479b96f38d291ebedaf" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:08.547453 28603 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.551923 24863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.552894 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.553902 24732 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.557176 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.563007 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.567114 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.580862 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.599707 24753 raft_consensus.cc:1261] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Refusing update from remote peer 34ee00b6f4554b39ab5e5d1a0252fae1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:00:08.600230 28820 consensus_queue.cc:1035] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4b1969054146119af3a958ef85de55" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:08.625227 28623 consensus_queue.cc:237] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:08.625880 28989 raft_consensus.cc:2932] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:08.628196 24607 catalog_manager.cc:5484] T ade0bed961d448449904f531c57ad940 P 34ee00b6f4554b39ab5e5d1a0252fae1 reported cstate change: config changed from index 2 to 4, VOTER 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130) evicted. New cstate: current_term: 2 leader_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:08.630671 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet ade0bed961d448449904f531c57ad940 with delete_type TABLET_DATA_TOMBSTONED (TS 6e4fcab25c864479b96f38d291ebedaf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48648
I20240502 14:00:08.631069 29131 tablet_replica.cc:330] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: stopping tablet replica
I20240502 14:00:08.631208 29131 raft_consensus.cc:2229] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:08.631343 29131 raft_consensus.cc:2258] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:08.632910 29131 ts_tablet_manager.cc:1875] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:08.635084 29131 ts_tablet_manager.cc:1888] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:00:08.635171 29131 log.cc:1196] T ade0bed961d448449904f531c57ad940 P 6e4fcab25c864479b96f38d291ebedaf: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/ade0bed961d448449904f531c57ad940
I20240502 14:00:08.635799 24595 catalog_manager.cc:4830] TS 6e4fcab25c864479b96f38d291ebedaf (127.22.55.130:38051): tablet ade0bed961d448449904f531c57ad940 (table TestTable [id=e67d2bd1c0d84039b6c44b58e82916af]) successfully deleted
I20240502 14:00:08.695063 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.696790 24732 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.698518 28603 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.704787 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.715037 24863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.715981 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.719579 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.722378 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:08.769519 28623 consensus_queue.cc:237] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } }
I20240502 14:00:08.770131 28989 raft_consensus.cc:2932] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:08.772456 24609 catalog_manager.cc:5484] T 75834b9995394a2d95a56acc95eea043 P 34ee00b6f4554b39ab5e5d1a0252fae1 reported cstate change: config changed from index 4 to 6, VOTER 7b4b1969054146119af3a958ef85de55 (127.22.55.129) evicted. New cstate: current_term: 2 leader_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "34ee00b6f4554b39ab5e5d1a0252fae1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 45809 } } }
I20240502 14:00:08.775122 24732 tablet_service.cc:1514] Processing DeleteTablet for tablet 75834b9995394a2d95a56acc95eea043 with delete_type TABLET_DATA_TOMBSTONED (TS 7b4b1969054146119af3a958ef85de55 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59640
I20240502 14:00:08.775506 29181 tablet_replica.cc:330] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: stopping tablet replica
I20240502 14:00:08.775638 29181 raft_consensus.cc:2229] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:08.775734 29181 raft_consensus.cc:2258] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:08.777002 29181 ts_tablet_manager.cc:1875] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:08.779268 29181 ts_tablet_manager.cc:1888] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:00:08.779348 29181 log.cc:1196] T 75834b9995394a2d95a56acc95eea043 P 7b4b1969054146119af3a958ef85de55: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/75834b9995394a2d95a56acc95eea043
I20240502 14:00:08.779920 24596 catalog_manager.cc:4830] TS 7b4b1969054146119af3a958ef85de55 (127.22.55.129:36649): tablet 75834b9995394a2d95a56acc95eea043 (table rebalance_test_table_4 [id=2a4b94a9885349e8bb805979c6d23fb0]) successfully deleted
I20240502 14:00:09.043120 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.045903 24732 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.059643 28603 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.060866 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.061275 24863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.072111 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.072254 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.089277 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.172565 24863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.174988 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.175248 28603 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.178465 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.179877 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.186270 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.189021 24732 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.199548 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.290510 25998 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.291468 28603 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.291848 24732 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.295429 26260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.302062 26391 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.315625 24863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.316526 25867 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:09.328325 26129 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
84497bd32f594599b786a42799ec8d7a | 127.22.55.190:38111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.55.148:41981 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
34ee00b6f4554b39ab5e5d1a0252fae1 | 127.22.55.136:45809 | HEALTHY | <none> | 2 | 0
51d1006508074f10acd579729457b0c5 | 127.22.55.131:42509 | HEALTHY | <none> | 2 | 0
6e4fcab25c864479b96f38d291ebedaf | 127.22.55.130:38051 | HEALTHY | <none> | 2 | 0
7b4b1969054146119af3a958ef85de55 | 127.22.55.129:36649 | HEALTHY | <none> | 2 | 0
8aa9c677a2c84c5082c708ae767eb3a8 | 127.22.55.132:39445 | HEALTHY | <none> | 2 | 0
a9e66f5a1b7f4202860a428d70203609 | 127.22.55.135:41681 | HEALTHY | <none> | 2 | 0
b2a245b6dbf34e03a431a50a7b54ce92 | 127.22.55.133:38561 | HEALTHY | <none> | 2 | 0
c970e31aa29e4944ab5fb3024984aefd | 127.22.55.134:41009 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.22.55.129 | experimental | 127.22.55.129:36649
local_ip_for_outbound_sockets | 127.22.55.130 | experimental | 127.22.55.130:38051
local_ip_for_outbound_sockets | 127.22.55.131 | experimental | 127.22.55.131:42509
local_ip_for_outbound_sockets | 127.22.55.132 | experimental | 127.22.55.132:39445
local_ip_for_outbound_sockets | 127.22.55.133 | experimental | 127.22.55.133:38561
local_ip_for_outbound_sockets | 127.22.55.134 | experimental | 127.22.55.134:41009
local_ip_for_outbound_sockets | 127.22.55.135 | experimental | 127.22.55.135:41681
local_ip_for_outbound_sockets | 127.22.55.136 | experimental | 127.22.55.136:45809
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.55.129:36649
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.55.130:38051
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.55.131:42509
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.55.132:39445
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.55.133:38561
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.55.134:41009
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.55.135:41681
server_dump_info_path | /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.22.55.136:45809
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.55.148:41981 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:00:09.349690 22750 log_verifier.cc:126] Checking tablet 00c8e8a7c7a6493088d08577906bf21c
I20240502 14:00:09.356091 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 00c8e8a7c7a6493088d08577906bf21c
I20240502 14:00:09.356153 22750 log_verifier.cc:126] Checking tablet 07dbaad8ae36426b9eff6ea62db21dc7
I20240502 14:00:09.361588 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 07dbaad8ae36426b9eff6ea62db21dc7
I20240502 14:00:09.361649 22750 log_verifier.cc:126] Checking tablet 0cad80f3ea304711b302663a1f420aba
I20240502 14:00:09.367146 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0cad80f3ea304711b302663a1f420aba
I20240502 14:00:09.367205 22750 log_verifier.cc:126] Checking tablet 1ae2eb2957c749b4a34b07944a5c3592
I20240502 14:00:09.372911 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1ae2eb2957c749b4a34b07944a5c3592
I20240502 14:00:09.372972 22750 log_verifier.cc:126] Checking tablet 27fa46c3e6214af2a67805b7dcfd9301
I20240502 14:00:09.378916 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 27fa46c3e6214af2a67805b7dcfd9301
I20240502 14:00:09.378975 22750 log_verifier.cc:126] Checking tablet 4fe45162fdb94d61961fea1cb7cc6af8
I20240502 14:00:09.385064 22750 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4fe45162fdb94d61961fea1cb7cc6af8
I20240502 14:00:09.385128 22750 log_verifier.cc:126] Checking tablet 6ca9e7a8cb7644f48a634b8ef86cad9f
I20240502 14:00:09.390812 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6ca9e7a8cb7644f48a634b8ef86cad9f
I20240502 14:00:09.390882 22750 log_verifier.cc:126] Checking tablet 75834b9995394a2d95a56acc95eea043
I20240502 14:00:09.399328 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 75834b9995394a2d95a56acc95eea043
I20240502 14:00:09.399420 22750 log_verifier.cc:126] Checking tablet 94738674b9af470f922b593daec77442
I20240502 14:00:09.405011 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 94738674b9af470f922b593daec77442
I20240502 14:00:09.405068 22750 log_verifier.cc:126] Checking tablet 94e5deba02294d4e85fd4bad586a78d5
I20240502 14:00:09.410712 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 94e5deba02294d4e85fd4bad586a78d5
I20240502 14:00:09.410766 22750 log_verifier.cc:126] Checking tablet ade0bed961d448449904f531c57ad940
I20240502 14:00:09.418036 22750 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ade0bed961d448449904f531c57ad940
I20240502 14:00:09.418108 22750 log_verifier.cc:126] Checking tablet b81678740ddc4fce9da067dc3d8dff8d
I20240502 14:00:09.424033 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b81678740ddc4fce9da067dc3d8dff8d
I20240502 14:00:09.424096 22750 log_verifier.cc:126] Checking tablet cae053fe22fd451e88e8e01eb93b76c5
I20240502 14:00:09.429702 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet cae053fe22fd451e88e8e01eb93b76c5
I20240502 14:00:09.429762 22750 log_verifier.cc:126] Checking tablet d465839425704eef962eb345979233f6
I20240502 14:00:09.435461 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d465839425704eef962eb345979233f6
I20240502 14:00:09.435521 22750 log_verifier.cc:126] Checking tablet f943154c90654cd39f602a0fc2b90ceb
I20240502 14:00:09.441131 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f943154c90654cd39f602a0fc2b90ceb
I20240502 14:00:09.441196 22750 log_verifier.cc:126] Checking tablet ff8c1d90d5d8449aac5e3e1e81d54523
I20240502 14:00:09.446882 22750 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ff8c1d90d5d8449aac5e3e1e81d54523
I20240502 14:00:09.447335 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 24671
I20240502 14:00:09.454402 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 24802
I20240502 14:00:09.461241 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 25805
I20240502 14:00:09.467170 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 25936
I20240502 14:00:09.473035 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 26067
I20240502 14:00:09.479743 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 26198
I20240502 14:00:09.485383 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 26329
I20240502 14:00:09.491205 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 28541
I20240502 14:00:09.496834 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 24580
2024-05-02T14:00:09Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (29162 ms)
[----------] 1 test from RebalanceParamTest (29162 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240502 14:00:09.537011 22750 test_util.cc:268] Using random seed: 90791406
I20240502 14:00:09.537783 22750 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:00:09.537822 22750 ts_itest-base.cc:113] --------------
I20240502 14:00:09.537833 22750 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:00:09.537843 22750 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:00:09.537854 22750 ts_itest-base.cc:116] --------------
2024-05-02T14:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:00:09Z Disabled control of system clock
I20240502 14:00:09.546203 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:32851
--webserver_interface=127.22.55.190
--webserver_port=0
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.55.190:32851
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 14:00:09.649542 29298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:09.649783 29298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:09.649883 29298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:09.652925 29298 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:09.652995 29298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:00:09.653036 29298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:09.653076 29298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:00:09.653111 29298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:00:09.656672 29298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.55.190:32851
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.55.190:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.55.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.29298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:09.657573 29298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:09.658695 29298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:09.666662 29306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:09.667600 29303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:09.668398 29304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:09.669384 29298 server_base.cc:1034] running on GCE node
I20240502 14:00:09.669916 29298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:09.670881 29298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:09.672040 29298 hybrid_clock.cc:648] HybridClock initialized: now 1714658409672023 us; error 33 us; skew 500 ppm
I20240502 14:00:09.673889 29298 webserver.cc:460] Webserver started at http://127.22.55.190:44707/ using document root <none> and password file <none>
I20240502 14:00:09.674419 29298 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:09.674471 29298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:09.674628 29298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:09.676272 29298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "091053291c28421292039b46a8cdf3ff"
format_stamp: "Formatted at 2024-05-02 14:00:09 on dist-test-slave-sdf5"
I20240502 14:00:09.676723 29298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "091053291c28421292039b46a8cdf3ff"
format_stamp: "Formatted at 2024-05-02 14:00:09 on dist-test-slave-sdf5"
I20240502 14:00:09.680016 29298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:00:09.682185 29312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:09.683219 29298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:09.683315 29298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
uuid: "091053291c28421292039b46a8cdf3ff"
format_stamp: "Formatted at 2024-05-02 14:00:09 on dist-test-slave-sdf5"
I20240502 14:00:09.683399 29298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:09.693678 29298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:09.694211 29298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:09.694339 29298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:09.700678 29298 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.190:32851
I20240502 14:00:09.700690 29364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.190:32851 every 8 connection(s)
I20240502 14:00:09.701643 29298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:00:09.704159 29365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:09.708922 29365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Bootstrap starting.
I20240502 14:00:09.711081 29365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:09.711638 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 29298
I20240502 14:00:09.711699 29365 log.cc:824] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:09.711752 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:00:09.713549 29365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: No bootstrap required, opened a new log
I20240502 14:00:09.716328 29365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091053291c28421292039b46a8cdf3ff" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 32851 } }
I20240502 14:00:09.716550 29365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:09.716598 29365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091053291c28421292039b46a8cdf3ff, State: Initialized, Role: FOLLOWER
I20240502 14:00:09.717047 29365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091053291c28421292039b46a8cdf3ff" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 32851 } }
I20240502 14:00:09.717222 29365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:09.717298 29365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:09.717422 29365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:09.718302 29365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091053291c28421292039b46a8cdf3ff" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 32851 } }
I20240502 14:00:09.718618 29365 leader_election.cc:302] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 091053291c28421292039b46a8cdf3ff; no voters:
I20240502 14:00:09.718930 29365 leader_election.cc:288] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:09.719305 29370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:09.719498 29370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [term 1 LEADER]: Becoming Leader. State: Replica: 091053291c28421292039b46a8cdf3ff, State: Running, Role: LEADER
I20240502 14:00:09.719792 29370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091053291c28421292039b46a8cdf3ff" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 32851 } }
I20240502 14:00:09.720085 29365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:09.721246 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "091053291c28421292039b46a8cdf3ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091053291c28421292039b46a8cdf3ff" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 32851 } } }
I20240502 14:00:09.721374 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:09.721586 29372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 091053291c28421292039b46a8cdf3ff. Latest consensus state: current_term: 1 leader_uuid: "091053291c28421292039b46a8cdf3ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091053291c28421292039b46a8cdf3ff" member_type: VOTER last_known_addr { host: "127.22.55.190" port: 32851 } } }
I20240502 14:00:09.721675 29372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:09.721735 29379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:09.724674 29379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:09.729947 29379 catalog_manager.cc:1347] Generated new cluster ID: fb117c446bce48a5a14f2550282ead59
I20240502 14:00:09.730021 29379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:09.747785 29379 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:09.749020 29379 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:09.757205 29379 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Generated new TSK 0
I20240502 14:00:09.758020 29379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:09.765960 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.129:0
--local_ip_for_outbound_sockets=127.22.55.129
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:09.873651 29389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:09.873876 29389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:09.873950 29389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:09.877009 29389 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:09.877087 29389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:09.877192 29389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.129
I20240502 14:00:09.880957 29389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.55.129
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.29389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.129
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:09.881907 29389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:09.883096 29389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:09.892160 29394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:09.893085 29397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:09.893886 29395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:09.894783 29389 server_base.cc:1034] running on GCE node
I20240502 14:00:09.895251 29389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:09.895834 29389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:09.896983 29389 hybrid_clock.cc:648] HybridClock initialized: now 1714658409896972 us; error 26 us; skew 500 ppm
I20240502 14:00:09.899000 29389 webserver.cc:460] Webserver started at http://127.22.55.129:36831/ using document root <none> and password file <none>
I20240502 14:00:09.899590 29389 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:09.899648 29389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:09.899838 29389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:09.901609 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1d00d3a47c52478894cf8c66959ef7bd"
format_stamp: "Formatted at 2024-05-02 14:00:09 on dist-test-slave-sdf5"
I20240502 14:00:09.902089 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1d00d3a47c52478894cf8c66959ef7bd"
format_stamp: "Formatted at 2024-05-02 14:00:09 on dist-test-slave-sdf5"
I20240502 14:00:09.905613 29389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20240502 14:00:09.907836 29403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:09.908843 29389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:09.908963 29389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1d00d3a47c52478894cf8c66959ef7bd"
format_stamp: "Formatted at 2024-05-02 14:00:09 on dist-test-slave-sdf5"
I20240502 14:00:09.909058 29389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:09.922490 29389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:09.923179 29389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:09.923348 29389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:09.923877 29389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:09.924758 29389 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:09.924808 29389 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:09.924866 29389 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:09.924899 29389 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:09.933337 29389 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.129:44823
I20240502 14:00:09.933378 29516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.129:44823 every 8 connection(s)
I20240502 14:00:09.934245 29389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:00:09.942082 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 29389
I20240502 14:00:09.942242 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:00:09.943130 29517 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:09.943348 29517 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:09.943871 29517 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:09.944368 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.130:0
--local_ip_for_outbound_sockets=127.22.55.130
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:09.945181 29329 ts_manager.cc:194] Registered new tserver with Master: 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:09.946465 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.129:55051
W20240502 14:00:10.056157 29520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:10.056411 29520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:10.056490 29520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:10.059593 29520 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:10.059682 29520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:10.059785 29520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.130
I20240502 14:00:10.063460 29520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.55.130
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.29520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.130
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:10.064430 29520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:10.065564 29520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:10.074478 29525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.075448 29528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.076222 29526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:10.077209 29520 server_base.cc:1034] running on GCE node
I20240502 14:00:10.077586 29520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:10.078125 29520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:10.079268 29520 hybrid_clock.cc:648] HybridClock initialized: now 1714658410079253 us; error 28 us; skew 500 ppm
I20240502 14:00:10.081071 29520 webserver.cc:460] Webserver started at http://127.22.55.130:34019/ using document root <none> and password file <none>
I20240502 14:00:10.081591 29520 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:10.081643 29520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:10.081799 29520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:10.083420 29520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d6e2e2c10def4331bec17bd110c9b423"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.083853 29520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d6e2e2c10def4331bec17bd110c9b423"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.087071 29520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:00:10.089215 29534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.090261 29520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:10.090358 29520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d6e2e2c10def4331bec17bd110c9b423"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.090432 29520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:10.105661 29520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:10.106263 29520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:10.106400 29520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:10.106950 29520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:10.107818 29520 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:10.107864 29520 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.107903 29520 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:10.107923 29520 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.116586 29520 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.130:40041
I20240502 14:00:10.116621 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.130:40041 every 8 connection(s)
I20240502 14:00:10.117484 29520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:00:10.120471 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 29520
I20240502 14:00:10.120579 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:00:10.122822 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.131:0
--local_ip_for_outbound_sockets=127.22.55.131
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:10.127089 29648 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:10.127310 29648 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:10.127739 29648 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:10.128695 29329 ts_manager.cc:194] Registered new tserver with Master: d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:10.129220 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.130:50985
W20240502 14:00:10.230664 29651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:10.230893 29651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:10.230938 29651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:10.233867 29651 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:10.233928 29651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:10.234027 29651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.131
I20240502 14:00:10.237466 29651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.55.131
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.29651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.131
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:10.238358 29651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:10.239503 29651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:10.248318 29656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.249259 29657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.250000 29659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:10.250986 29651 server_base.cc:1034] running on GCE node
I20240502 14:00:10.251371 29651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:10.251935 29651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:10.253103 29651 hybrid_clock.cc:648] HybridClock initialized: now 1714658410253076 us; error 47 us; skew 500 ppm
I20240502 14:00:10.255059 29651 webserver.cc:460] Webserver started at http://127.22.55.131:37029/ using document root <none> and password file <none>
I20240502 14:00:10.255620 29651 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:10.255697 29651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:10.255867 29651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:10.257907 29651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.258519 29651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.262055 29651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 14:00:10.264333 29665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.265328 29651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:10.265527 29651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.265628 29651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:10.278362 29651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:10.278990 29651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:10.279166 29651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:10.279698 29651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:10.280539 29651 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:10.280586 29651 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.280656 29651 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:10.280687 29651 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.288911 29651 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.131:38105
I20240502 14:00:10.288947 29778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.131:38105 every 8 connection(s)
I20240502 14:00:10.289816 29651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:00:10.298161 29779 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:10.298360 29779 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:10.298759 29779 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:10.299098 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 29651
I20240502 14:00:10.299208 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:00:10.299747 29329 ts_manager.cc:194] Registered new tserver with Master: efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:10.300410 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.131:42903
I20240502 14:00:10.301383 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.132:0
--local_ip_for_outbound_sockets=127.22.55.132
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:10.408254 29782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:10.408473 29782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:10.408516 29782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:10.411449 29782 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:10.411506 29782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:10.411571 29782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.132
I20240502 14:00:10.415120 29782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.55.132
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.29782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.132
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:10.415977 29782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:10.417093 29782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:10.425487 29788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.426231 29790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.426800 29787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:10.427578 29782 server_base.cc:1034] running on GCE node
I20240502 14:00:10.427959 29782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:10.428493 29782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:10.429643 29782 hybrid_clock.cc:648] HybridClock initialized: now 1714658410429625 us; error 35 us; skew 500 ppm
I20240502 14:00:10.431655 29782 webserver.cc:460] Webserver started at http://127.22.55.132:40849/ using document root <none> and password file <none>
I20240502 14:00:10.432228 29782 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:10.432284 29782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:10.432471 29782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:10.434087 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.434594 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.438014 29782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 14:00:10.440248 29796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.441220 29782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:10.441316 29782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.441411 29782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:10.468190 29782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:10.468837 29782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:10.468999 29782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:10.469563 29782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:10.470496 29782 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:10.470547 29782 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.470604 29782 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:10.470639 29782 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.479089 29782 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.132:44915
I20240502 14:00:10.479130 29909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.132:44915 every 8 connection(s)
I20240502 14:00:10.480037 29782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:00:10.487586 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 29782
I20240502 14:00:10.487725 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:00:10.488373 29910 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:10.488590 29910 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:10.489074 29910 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:10.489764 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.133:0
--local_ip_for_outbound_sockets=127.22.55.133
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:10.489959 29329 ts_manager.cc:194] Registered new tserver with Master: 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:10.490460 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.132:56099
W20240502 14:00:10.595786 29913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:10.596028 29913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:10.596101 29913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:10.599098 29913 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:10.599174 29913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:10.599269 29913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.133
I20240502 14:00:10.602759 29913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.55.133
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.29913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.133
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:10.603711 29913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:10.604826 29913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:10.613490 29918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.614063 29921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.614594 29919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:10.615371 29913 server_base.cc:1034] running on GCE node
I20240502 14:00:10.615734 29913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:10.616278 29913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:10.617429 29913 hybrid_clock.cc:648] HybridClock initialized: now 1714658410617406 us; error 40 us; skew 500 ppm
I20240502 14:00:10.619467 29913 webserver.cc:460] Webserver started at http://127.22.55.133:37223/ using document root <none> and password file <none>
I20240502 14:00:10.620052 29913 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:10.620113 29913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:10.620296 29913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:10.622131 29913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.622642 29913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.626093 29913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:10.628422 29927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.629434 29913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:10.629531 29913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.629626 29913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:10.640761 29913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:10.641412 29913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:10.641577 29913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:10.642117 29913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:10.643076 29913 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:10.643133 29913 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.643193 29913 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:10.643218 29913 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.651809 29913 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.133:39647
I20240502 14:00:10.651858 30040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.133:39647 every 8 connection(s)
I20240502 14:00:10.652742 29913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:10.655875 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 29913
I20240502 14:00:10.656010 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:00:10.658347 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.134:0
--local_ip_for_outbound_sockets=127.22.55.134
--webserver_interface=127.22.55.134
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:10.662530 30041 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:10.662750 30041 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:10.663233 30041 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:10.664299 29329 ts_manager.cc:194] Registered new tserver with Master: f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:10.665144 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.133:44193
W20240502 14:00:10.766541 30044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:10.766765 30044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:10.766809 30044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:10.770104 30044 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:10.770179 30044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:10.770251 30044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.134
I20240502 14:00:10.773959 30044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.55.134
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.30044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.134
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:10.774880 30044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:10.776008 30044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:10.784585 30050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.785030 30049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.785781 30052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:10.786446 30044 server_base.cc:1034] running on GCE node
I20240502 14:00:10.786837 30044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:10.787463 30044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:10.788612 30044 hybrid_clock.cc:648] HybridClock initialized: now 1714658410788599 us; error 27 us; skew 500 ppm
I20240502 14:00:10.790392 30044 webserver.cc:460] Webserver started at http://127.22.55.134:45535/ using document root <none> and password file <none>
I20240502 14:00:10.790974 30044 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:10.791033 30044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:10.791214 30044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:10.792995 30044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.794967 30044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.798552 30044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:00:10.800858 30058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.801904 30044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:10.802012 30044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.802114 30044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:10.814141 30044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:10.814786 30044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:10.814965 30044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:10.815503 30044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:10.816403 30044 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:10.816450 30044 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.816493 30044 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:10.816525 30044 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.825219 30044 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.134:44137
I20240502 14:00:10.825263 30171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.134:44137 every 8 connection(s)
I20240502 14:00:10.826160 30044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:00:10.834899 30172 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:10.834928 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 30044
I20240502 14:00:10.835063 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:00:10.835182 30172 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:10.835760 30172 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:10.836843 29329 ts_manager.cc:194] Registered new tserver with Master: 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:10.837405 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.135:0
--local_ip_for_outbound_sockets=127.22.55.135
--webserver_interface=127.22.55.135
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:10.837621 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.134:39417
I20240502 14:00:10.948549 29517 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
W20240502 14:00:10.948872 30175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:10.949123 30175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:10.949223 30175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:10.952411 30175 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:10.952477 30175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:10.952572 30175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.135
I20240502 14:00:10.956245 30175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.55.135
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.30175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.135
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:10.957197 30175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:10.958369 30175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:10.967226 30180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.967676 30181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:10.968245 30183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:10.969106 30175 server_base.cc:1034] running on GCE node
I20240502 14:00:10.969489 30175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:10.970126 30175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:10.971290 30175 hybrid_clock.cc:648] HybridClock initialized: now 1714658410971277 us; error 31 us; skew 500 ppm
I20240502 14:00:10.973393 30175 webserver.cc:460] Webserver started at http://127.22.55.135:39467/ using document root <none> and password file <none>
I20240502 14:00:10.973969 30175 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:10.974037 30175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:10.974226 30175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:10.975988 30175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.976490 30175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.979892 30175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:00:10.982224 30189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.983289 30175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:00:10.983409 30175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:10.983505 30175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:10.997285 30175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:10.998024 30175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:10.998222 30175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:10.998795 30175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:10.999828 30175 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:10.999879 30175 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:10.999941 30175 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:10.999975 30175 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:11.008929 30175 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.135:40995
I20240502 14:00:11.008952 30302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.135:40995 every 8 connection(s)
I20240502 14:00:11.009963 30175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:00:11.013955 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 30175
I20240502 14:00:11.014086 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:00:11.016465 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.136:0
--local_ip_for_outbound_sockets=127.22.55.136
--webserver_interface=127.22.55.136
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:11.019658 30303 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:11.019897 30303 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:11.020399 30303 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:11.021477 29329 ts_manager.cc:194] Registered new tserver with Master: f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:11.022404 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.135:56789
I20240502 14:00:11.131093 29648 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
W20240502 14:00:11.137434 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:11.137667 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:11.137712 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:11.140970 30306 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:11.141042 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:11.141111 30306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.136
I20240502 14:00:11.144758 30306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.55.136
--webserver_port=0
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.30306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.136
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:11.145701 30306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:11.146865 30306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:11.155413 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:11.156006 30314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:11.156595 30311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:11.157577 30306 server_base.cc:1034] running on GCE node
I20240502 14:00:11.157954 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:11.158504 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:11.159651 30306 hybrid_clock.cc:648] HybridClock initialized: now 1714658411159623 us; error 42 us; skew 500 ppm
I20240502 14:00:11.161556 30306 webserver.cc:460] Webserver started at http://127.22.55.136:46663/ using document root <none> and password file <none>
I20240502 14:00:11.162128 30306 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:11.162191 30306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:11.162379 30306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:11.164050 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "8973f2527acb48f5ad36589652de73e1"
format_stamp: "Formatted at 2024-05-02 14:00:11 on dist-test-slave-sdf5"
I20240502 14:00:11.164502 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "8973f2527acb48f5ad36589652de73e1"
format_stamp: "Formatted at 2024-05-02 14:00:11 on dist-test-slave-sdf5"
I20240502 14:00:11.167726 30306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:00:11.170171 30320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:11.171233 30306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:00:11.171329 30306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8973f2527acb48f5ad36589652de73e1"
format_stamp: "Formatted at 2024-05-02 14:00:11 on dist-test-slave-sdf5"
I20240502 14:00:11.171416 30306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:11.201719 30306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:11.202400 30306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:11.202545 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:11.203152 30306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:11.204046 30306 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:11.204090 30306 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:11.204130 30306 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:11.204150 30306 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:11.212663 30306 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.136:44201
I20240502 14:00:11.212705 30433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.136:44201 every 8 connection(s)
I20240502 14:00:11.213717 30306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:00:11.222565 30434 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:11.222816 30434 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:11.223326 30434 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:11.223421 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 30306
I20240502 14:00:11.223546 22750 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:00:11.224393 29329 ts_manager.cc:194] Registered new tserver with Master: 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:11.224932 29329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.136:39595
I20240502 14:00:11.225878 22750 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:00:11.235406 22750 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:00:11.235482 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 29913
I20240502 14:00:11.241464 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 30044
I20240502 14:00:11.247082 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 30175
I20240502 14:00:11.253109 22750 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCplTwR/build/debug/bin/kudu with pid 30306
I20240502 14:00:11.302296 29779 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
I20240502 14:00:11.492244 29910 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
I20240502 14:00:15.012876 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:15.034265 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 67a3a069498247aea44c04fe6ec1860d (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.034265 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 67a3a069498247aea44c04fe6ec1860d (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.035349 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a3a069498247aea44c04fe6ec1860d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.035342 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a3a069498247aea44c04fe6ec1860d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.034265 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 5548ad03819446c88aaa61540ffb193d (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.036092 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5548ad03819446c88aaa61540ffb193d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.034265 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 7345a902a3754d4a98107d3a968f6eb1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.037634 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 7345a902a3754d4a98107d3a968f6eb1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.038132 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345a902a3754d4a98107d3a968f6eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.038038 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 7345a902a3754d4a98107d3a968f6eb1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.038583 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345a902a3754d4a98107d3a968f6eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.038866 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345a902a3754d4a98107d3a968f6eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.039214 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 67a3a069498247aea44c04fe6ec1860d (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.039516 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a3a069498247aea44c04fe6ec1860d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.042266 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 5548ad03819446c88aaa61540ffb193d (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.042560 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5548ad03819446c88aaa61540ffb193d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.037642 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 5548ad03819446c88aaa61540ffb193d (DEFAULT_TABLE table=rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.043251 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5548ad03819446c88aaa61540ffb193d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.045711 30455 tablet_bootstrap.cc:492] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:15.047750 30454 tablet_bootstrap.cc:492] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:15.048102 30455 tablet_bootstrap.cc:654] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.048936 30455 log.cc:824] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:15.049731 30457 tablet_bootstrap.cc:492] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:15.050331 30456 tablet_bootstrap.cc:492] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:15.050380 30454 tablet_bootstrap.cc:654] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.051208 30454 log.cc:824] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:15.052067 30457 tablet_bootstrap.cc:654] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.052296 30456 tablet_bootstrap.cc:654] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.052829 30457 log.cc:824] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:15.052902 30456 log.cc:824] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:15.053220 30454 tablet_bootstrap.cc:492] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:15.053423 30454 ts_tablet_manager.cc:1367] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:00:15.054252 30455 tablet_bootstrap.cc:492] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:15.054435 30455 ts_tablet_manager.cc:1367] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:00:15.054479 30457 tablet_bootstrap.cc:492] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:15.054633 30457 ts_tablet_manager.cc:1367] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:00:15.055733 30456 tablet_bootstrap.cc:492] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:15.055940 30456 ts_tablet_manager.cc:1367] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 14:00:15.057358 30454 raft_consensus.cc:348] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.057585 30454 raft_consensus.cc:374] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.057646 30454 raft_consensus.cc:729] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.057991 30455 raft_consensus.cc:348] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.058239 30455 raft_consensus.cc:374] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.058161 30454 consensus_queue.cc:260] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.058300 30455 raft_consensus.cc:729] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.058324 30457 raft_consensus.cc:348] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.058516 30457 raft_consensus.cc:374] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.058571 30457 raft_consensus.cc:729] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.059008 30457 consensus_queue.cc:260] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.059105 30454 ts_tablet_manager.cc:1398] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:00:15.059492 30454 tablet_bootstrap.cc:492] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:15.059826 30457 ts_tablet_manager.cc:1398] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 14:00:15.059875 30455 consensus_queue.cc:260] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.060484 30457 tablet_bootstrap.cc:492] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:15.060631 30454 tablet_bootstrap.cc:654] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.061514 30457 tablet_bootstrap.cc:654] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.061439 30456 raft_consensus.cc:348] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.061741 30456 raft_consensus.cc:374] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.061833 30456 raft_consensus.cc:729] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.061879 30454 tablet_bootstrap.cc:492] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:15.061975 30454 ts_tablet_manager.cc:1367] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:15.062371 30456 consensus_queue.cc:260] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.062594 30454 raft_consensus.cc:348] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.062703 30454 raft_consensus.cc:374] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.062747 30454 raft_consensus.cc:729] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.062815 30457 tablet_bootstrap.cc:492] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:15.062928 30457 ts_tablet_manager.cc:1367] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:15.062907 30454 consensus_queue.cc:260] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.063257 30454 ts_tablet_manager.cc:1398] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:15.063493 30457 raft_consensus.cc:348] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.063608 30457 raft_consensus.cc:374] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.063649 30457 raft_consensus.cc:729] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.063727 30456 ts_tablet_manager.cc:1398] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240502 14:00:15.063786 30457 consensus_queue.cc:260] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.064105 30456 tablet_bootstrap.cc:492] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:15.065119 30456 tablet_bootstrap.cc:654] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.066381 30457 ts_tablet_manager.cc:1398] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:15.066417 30456 tablet_bootstrap.cc:492] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:15.066496 30456 ts_tablet_manager.cc:1367] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:15.067050 30456 raft_consensus.cc:348] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.067162 30456 raft_consensus.cc:374] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.067201 30456 raft_consensus.cc:729] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.067318 30456 consensus_queue.cc:260] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.067559 30456 ts_tablet_manager.cc:1398] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:15.067565 30455 ts_tablet_manager.cc:1398] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:00:15.067795 30455 tablet_bootstrap.cc:492] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:15.069126 30455 tablet_bootstrap.cc:654] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.070480 30455 tablet_bootstrap.cc:492] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:15.070564 30455 ts_tablet_manager.cc:1367] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:15.071130 30455 raft_consensus.cc:348] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.071261 30455 raft_consensus.cc:374] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.071316 30455 raft_consensus.cc:729] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.071460 30455 consensus_queue.cc:260] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.071720 30455 ts_tablet_manager.cc:1398] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:15.071864 30455 tablet_bootstrap.cc:492] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:15.072721 30455 tablet_bootstrap.cc:654] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.073877 30455 tablet_bootstrap.cc:492] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:15.073990 30455 ts_tablet_manager.cc:1367] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:15.074393 30455 raft_consensus.cc:348] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.074466 30455 raft_consensus.cc:374] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.074486 30455 raft_consensus.cc:729] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.074594 30455 consensus_queue.cc:260] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.074893 30455 ts_tablet_manager.cc:1398] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 14:00:15.121905 29649 tablet.cc:2369] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:00:15.189210 29518 tablet.cc:2369] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:15.195194 30468 raft_consensus.cc:482] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:15.195381 30468 raft_consensus.cc:504] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.196447 30468 leader_election.cc:288] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:15.200475 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a3a069498247aea44c04fe6ec1860d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:15.200469 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a3a069498247aea44c04fe6ec1860d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:15.200803 29733 raft_consensus.cc:2454] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:15.200803 29602 raft_consensus.cc:2454] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:15.201256 29404 leader_election.cc:302] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:15.201524 30468 raft_consensus.cc:2781] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:15.201604 30468 raft_consensus.cc:482] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:15.201663 30468 raft_consensus.cc:3037] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.202558 30468 raft_consensus.cc:504] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.202953 30468 leader_election.cc:288] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:15.203286 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a3a069498247aea44c04fe6ec1860d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:15.203428 29602 raft_consensus.cc:3037] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.203460 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a3a069498247aea44c04fe6ec1860d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:15.203588 29733 raft_consensus.cc:3037] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.204459 29602 raft_consensus.cc:2454] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:15.204468 29733 raft_consensus.cc:2454] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:15.204761 29405 leader_election.cc:302] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:15.204928 30468 raft_consensus.cc:2781] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:15.205144 30468 raft_consensus.cc:686] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:15.205475 30468 consensus_queue.cc:237] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:15.208415 29327 catalog_manager.cc:5484] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:15.233141 30466 raft_consensus.cc:482] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:15.233282 30466 raft_consensus.cc:504] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.234355 30466 leader_election.cc:288] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
W20240502 14:00:15.234709 29911 tablet.cc:2369] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:15.239158 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345a902a3754d4a98107d3a968f6eb1" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:15.239362 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345a902a3754d4a98107d3a968f6eb1" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:15.239454 29864 raft_consensus.cc:2454] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:15.239567 29471 raft_consensus.cc:2454] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:15.239885 29667 leader_election.cc:302] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:15.240157 30466 raft_consensus.cc:2781] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:15.240228 30466 raft_consensus.cc:482] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:15.240272 30466 raft_consensus.cc:3037] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.241082 30466 raft_consensus.cc:504] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.241429 30466 leader_election.cc:288] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:15.241732 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345a902a3754d4a98107d3a968f6eb1" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:15.241776 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345a902a3754d4a98107d3a968f6eb1" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:15.241847 29471 raft_consensus.cc:3037] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.241878 29864 raft_consensus.cc:3037] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.242687 29471 raft_consensus.cc:2454] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:15.242926 29864 raft_consensus.cc:2454] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:15.243026 29668 leader_election.cc:302] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:15.243333 30466 raft_consensus.cc:2781] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:15.243530 30466 raft_consensus.cc:686] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:15.243875 30466 consensus_queue.cc:237] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.247406 29327 catalog_manager.cc:5484] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:00:15.295719 29780 tablet.cc:2369] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:15.371322 30463 raft_consensus.cc:482] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:15.371480 30463 raft_consensus.cc:504] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.372532 30463 leader_election.cc:288] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:15.376264 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548ad03819446c88aaa61540ffb193d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:15.376441 29471 raft_consensus.cc:2454] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:15.376610 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548ad03819446c88aaa61540ffb193d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:15.376750 29602 raft_consensus.cc:2454] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:15.376816 29799 leader_election.cc:302] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:15.377065 30463 raft_consensus.cc:2781] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:15.377135 30463 raft_consensus.cc:482] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:15.377189 30463 raft_consensus.cc:3037] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.378002 30463 raft_consensus.cc:504] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.378338 30463 leader_election.cc:288] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:15.378674 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548ad03819446c88aaa61540ffb193d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:15.378698 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548ad03819446c88aaa61540ffb193d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:15.378806 29602 raft_consensus.cc:3037] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.378808 29471 raft_consensus.cc:3037] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.379606 29602 raft_consensus.cc:2454] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:15.379630 29471 raft_consensus.cc:2454] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:15.379880 29797 leader_election.cc:302] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:15.380071 30463 raft_consensus.cc:2781] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:15.380302 30463 raft_consensus.cc:686] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:15.380606 30463 consensus_queue.cc:237] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:15.383373 29327 catalog_manager.cc:5484] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
W20240502 14:00:15.507912 30484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:15.508177 30484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:15.511363 30484 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:00:15.598500 29602 raft_consensus.cc:1261] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:15.598645 29471 raft_consensus.cc:1261] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:15.599323 30463 consensus_queue.cc:1035] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:15.599503 30482 consensus_queue.cc:1035] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:15.603300 29733 raft_consensus.cc:1261] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:15.604542 29601 raft_consensus.cc:1261] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:15.605031 30468 consensus_queue.cc:1035] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:15.605760 30512 mvcc.cc:205] Tried to move back new op lower bound from 7023240870284062720 to 7023240869401370624. Current Snapshot: MvccSnapshot[applied={T|T < 7023240870284062720}]
I20240502 14:00:15.606770 29864 raft_consensus.cc:1261] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:15.607014 30468 consensus_queue.cc:1035] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:15.607318 30466 consensus_queue.cc:1035] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:15.607620 30514 mvcc.cc:205] Tried to move back new op lower bound from 7023240870284062720 to 7023240869401370624. Current Snapshot: MvccSnapshot[applied={T|T < 7023240870284062720}]
I20240502 14:00:15.612063 29471 raft_consensus.cc:1261] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:15.612718 30478 consensus_queue.cc:1035] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:15.619105 30516 mvcc.cc:205] Tried to move back new op lower bound from 7023240870318039040 to 7023240868841496576. Current Snapshot: MvccSnapshot[applied={T|T < 7023240870318039040}]
I20240502 14:00:15.650444 30520 mvcc.cc:205] Tried to move back new op lower bound from 7023240870284062720 to 7023240869401370624. Current Snapshot: MvccSnapshot[applied={T|T < 7023240870284062720}]
INSERT report
rows total: 2000
time total: 102.515 ms
time per row: 0.0512573 ms
I20240502 14:00:15.667969 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:15.673555 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 59f4faff46704ca1837edd139596d9e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.673544 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 59f4faff46704ca1837edd139596d9e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.673689 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 59f4faff46704ca1837edd139596d9e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.673861 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f4faff46704ca1837edd139596d9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.673861 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f4faff46704ca1837edd139596d9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.673965 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f4faff46704ca1837edd139596d9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.675873 30542 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:15.676419 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 86808085d86746a9bea7fe0ea0647d5f (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.676710 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86808085d86746a9bea7fe0ea0647d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.676880 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2ab1ded2c241038d5685678aa819a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.677047 30542 tablet_bootstrap.cc:654] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.677124 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2ab1ded2c241038d5685678aa819a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.678149 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 86808085d86746a9bea7fe0ea0647d5f (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.678369 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86808085d86746a9bea7fe0ea0647d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.678484 30543 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:15.679205 30546 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:15.679234 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 86808085d86746a9bea7fe0ea0647d5f (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.679468 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86808085d86746a9bea7fe0ea0647d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.679636 30544 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:15.680065 30546 tablet_bootstrap.cc:654] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.680524 30544 tablet_bootstrap.cc:654] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.680565 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2ab1ded2c241038d5685678aa819a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.680753 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2ab1ded2c241038d5685678aa819a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.681648 30544 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:15.681650 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2ab1ded2c241038d5685678aa819a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:15.681732 30544 ts_tablet_manager.cc:1367] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:15.681887 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2ab1ded2c241038d5685678aa819a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:15.679483 30543 tablet_bootstrap.cc:654] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.682523 30544 raft_consensus.cc:348] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.682631 30544 raft_consensus.cc:374] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.682664 30544 raft_consensus.cc:729] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.682828 30544 consensus_queue.cc:260] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.683200 30543 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:15.683250 30544 ts_tablet_manager.cc:1398] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:15.683281 30543 ts_tablet_manager.cc:1367] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:15.683403 30544 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:15.683737 30543 raft_consensus.cc:348] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.683828 30543 raft_consensus.cc:374] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.683862 30543 raft_consensus.cc:729] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.683974 30543 consensus_queue.cc:260] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.684247 30543 ts_tablet_manager.cc:1398] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:15.684267 30544 tablet_bootstrap.cc:654] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.684433 30543 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:15.684798 30542 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:15.684881 30542 ts_tablet_manager.cc:1367] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:15.685331 30543 tablet_bootstrap.cc:654] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.685665 30542 raft_consensus.cc:348] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.685762 30542 raft_consensus.cc:374] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.685794 30542 raft_consensus.cc:729] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.685905 30542 consensus_queue.cc:260] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.686194 30542 ts_tablet_manager.cc:1398] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:15.686362 30542 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:15.686403 30543 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:15.686479 30543 ts_tablet_manager.cc:1367] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:15.686838 30544 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:15.686992 30544 ts_tablet_manager.cc:1367] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:15.686935 30543 raft_consensus.cc:348] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.687043 30543 raft_consensus.cc:374] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.687077 30543 raft_consensus.cc:729] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.687198 30543 consensus_queue.cc:260] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.687439 30543 ts_tablet_manager.cc:1398] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:15.687429 30544 raft_consensus.cc:348] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.687521 30544 raft_consensus.cc:374] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.687554 30544 raft_consensus.cc:729] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.687603 30543 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:15.687667 30544 consensus_queue.cc:260] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.687898 30544 ts_tablet_manager.cc:1398] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:15.688524 30542 tablet_bootstrap.cc:654] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.688827 30543 tablet_bootstrap.cc:654] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.689350 30546 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:15.689448 30546 ts_tablet_manager.cc:1367] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:15.689726 30542 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:15.689795 30542 ts_tablet_manager.cc:1367] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:15.689919 30546 raft_consensus.cc:348] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.690029 30546 raft_consensus.cc:374] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.690068 30546 raft_consensus.cc:729] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.690205 30546 consensus_queue.cc:260] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.690246 30542 raft_consensus.cc:348] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.690348 30542 raft_consensus.cc:374] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.690384 30542 raft_consensus.cc:729] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.690462 30546 ts_tablet_manager.cc:1398] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:15.690487 30542 consensus_queue.cc:260] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.690755 30542 ts_tablet_manager.cc:1398] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:15.690937 30542 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:15.691803 30542 tablet_bootstrap.cc:654] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:15.692309 30543 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:15.692680 30543 ts_tablet_manager.cc:1367] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:15.692879 30542 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:15.692966 30542 ts_tablet_manager.cc:1367] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:15.693362 30543 raft_consensus.cc:348] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.693480 30543 raft_consensus.cc:374] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.693444 30542 raft_consensus.cc:348] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.693521 30543 raft_consensus.cc:729] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.693531 30542 raft_consensus.cc:374] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:15.693563 30542 raft_consensus.cc:729] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:15.693646 30543 consensus_queue.cc:260] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.693663 30542 consensus_queue.cc:260] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.693897 30543 ts_tablet_manager.cc:1398] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:15.693902 30542 ts_tablet_manager.cc:1398] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:15.716187 30463 raft_consensus.cc:482] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:15.716308 30463 raft_consensus.cc:504] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.716764 30463 leader_election.cc:288] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:15.717165 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f4faff46704ca1837edd139596d9e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:15.717308 29602 raft_consensus.cc:2454] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:15.717590 29797 leader_election.cc:302] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:15.717772 30463 raft_consensus.cc:2781] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:15.717835 30463 raft_consensus.cc:482] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:15.717880 30463 raft_consensus.cc:3037] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.718933 30463 raft_consensus.cc:504] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.719281 30463 leader_election.cc:288] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:15.719601 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f4faff46704ca1837edd139596d9e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:15.719727 29602 raft_consensus.cc:3037] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.720568 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f4faff46704ca1837edd139596d9e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:15.720639 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f4faff46704ca1837edd139596d9e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:15.720703 29602 raft_consensus.cc:2454] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:15.720731 29733 raft_consensus.cc:2454] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:15.720994 29797 leader_election.cc:302] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:15.721174 30463 raft_consensus.cc:2781] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:15.721285 30463 raft_consensus.cc:686] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:15.721412 30463 consensus_queue.cc:237] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.723066 29323 catalog_manager.cc:5484] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:15.850276 30517 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:15.850431 30517 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.851481 30517 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:15.855214 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2ab1ded2c241038d5685678aa819a8" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:15.855214 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2ab1ded2c241038d5685678aa819a8" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:15.855379 29732 raft_consensus.cc:2454] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:15.855388 29864 raft_consensus.cc:2454] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:15.855712 29536 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:15.855940 30517 raft_consensus.cc:2781] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:15.856004 30517 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:15.856047 30517 raft_consensus.cc:3037] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.859391 30517 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.859764 30517 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:15.860204 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2ab1ded2c241038d5685678aa819a8" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:15.860203 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2ab1ded2c241038d5685678aa819a8" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:15.860337 29864 raft_consensus.cc:3037] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.860337 29732 raft_consensus.cc:3037] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.861307 29864 raft_consensus.cc:2454] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:15.861307 29732 raft_consensus.cc:2454] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:15.861636 29536 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:15.861802 30517 raft_consensus.cc:2781] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:15.862005 30517 raft_consensus.cc:686] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:15.862298 30517 consensus_queue.cc:237] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:15.864480 29323 catalog_manager.cc:5484] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:15.918331 30474 raft_consensus.cc:482] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:15.918489 30474 raft_consensus.cc:504] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.919032 30474 leader_election.cc:288] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:15.919441 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86808085d86746a9bea7fe0ea0647d5f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:15.919476 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86808085d86746a9bea7fe0ea0647d5f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:15.919572 29732 raft_consensus.cc:2454] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:15.919634 29602 raft_consensus.cc:2454] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:15.919875 29405 leader_election.cc:302] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:15.920047 30474 raft_consensus.cc:2781] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:15.920125 30474 raft_consensus.cc:482] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:15.920171 30474 raft_consensus.cc:3037] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.921039 30474 raft_consensus.cc:504] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.921382 30474 leader_election.cc:288] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:15.921694 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86808085d86746a9bea7fe0ea0647d5f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:15.921720 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86808085d86746a9bea7fe0ea0647d5f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:15.921804 29602 raft_consensus.cc:3037] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.921837 29732 raft_consensus.cc:3037] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:15.922602 29732 raft_consensus.cc:2454] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:15.922658 29602 raft_consensus.cc:2454] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:15.922896 29405 leader_election.cc:302] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:15.923050 30474 raft_consensus.cc:2781] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:15.923174 30474 raft_consensus.cc:686] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:15.923318 30474 consensus_queue.cc:237] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:15.924829 29323 catalog_manager.cc:5484] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:16.090627 30561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:16.090940 30561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:16.094025 30561 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:00:16.105437 29732 raft_consensus.cc:3037] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.106783 30482 consensus_queue.cc:1035] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.110170 30482 consensus_queue.cc:1035] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 14:00:16.188472 29864 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.189302 29602 raft_consensus.cc:1261] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.189318 30517 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.189970 30474 consensus_queue.cc:1035] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.191627 29732 raft_consensus.cc:1261] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.192941 30474 consensus_queue.cc:1035] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.200191 29732 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.200634 30558 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.065 ms
time per row: 0.0575323 ms
I20240502 14:00:16.255697 29324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:16.261014 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 65cceaa60c7e470ba8b8d205d026625b (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.261073 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 65cceaa60c7e470ba8b8d205d026625b (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.261353 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cceaa60c7e470ba8b8d205d026625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.261408 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cceaa60c7e470ba8b8d205d026625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.261775 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 58a2db3a7fca4420a94891cd96c54ec3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.262028 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a2db3a7fca4420a94891cd96c54ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.261996 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 58a2db3a7fca4420a94891cd96c54ec3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.262210 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a2db3a7fca4420a94891cd96c54ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.263809 30623 tablet_bootstrap.cc:492] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:16.263902 30622 tablet_bootstrap.cc:492] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:16.264461 29450 tablet_service.cc:1467] Processing CreateTablet for tablet bd9618b854814e11a3c6ae7504821d35 (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.264576 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 65cceaa60c7e470ba8b8d205d026625b (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.264693 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9618b854814e11a3c6ae7504821d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.264777 30623 tablet_bootstrap.cc:654] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.264815 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cceaa60c7e470ba8b8d205d026625b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.264814 30622 tablet_bootstrap.cc:654] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.266544 29843 tablet_service.cc:1467] Processing CreateTablet for tablet bd9618b854814e11a3c6ae7504821d35 (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.266770 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 58a2db3a7fca4420a94891cd96c54ec3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.266786 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9618b854814e11a3c6ae7504821d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.267195 29713 tablet_service.cc:1467] Processing CreateTablet for tablet bd9618b854814e11a3c6ae7504821d35 (DEFAULT_TABLE table=rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.267277 30622 tablet_bootstrap.cc:492] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:16.267372 30622 ts_tablet_manager.cc:1367] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:16.267403 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a2db3a7fca4420a94891cd96c54ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.267482 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9618b854814e11a3c6ae7504821d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.267820 30622 raft_consensus.cc:348] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.267913 30622 raft_consensus.cc:374] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.267940 30622 raft_consensus.cc:729] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.268083 30622 consensus_queue.cc:260] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.268388 30622 ts_tablet_manager.cc:1398] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.268643 30624 tablet_bootstrap.cc:492] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:16.269663 30627 tablet_bootstrap.cc:492] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:16.269730 30622 tablet_bootstrap.cc:492] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:16.270112 30624 tablet_bootstrap.cc:654] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.270650 30622 tablet_bootstrap.cc:654] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.270712 30627 tablet_bootstrap.cc:654] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.271287 30623 tablet_bootstrap.cc:492] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:16.271399 30623 ts_tablet_manager.cc:1367] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:16.271816 30622 tablet_bootstrap.cc:492] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:16.271898 30622 ts_tablet_manager.cc:1367] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:16.271908 30623 raft_consensus.cc:348] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.272025 30623 raft_consensus.cc:374] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.272058 30623 raft_consensus.cc:729] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.272188 30623 consensus_queue.cc:260] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.272326 30624 tablet_bootstrap.cc:492] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:16.272399 30624 ts_tablet_manager.cc:1367] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:16.272353 30622 raft_consensus.cc:348] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.272439 30622 raft_consensus.cc:374] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.272468 30623 ts_tablet_manager.cc:1398] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.272476 30622 raft_consensus.cc:729] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.272634 30623 tablet_bootstrap.cc:492] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:16.272604 30622 consensus_queue.cc:260] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.272854 30622 ts_tablet_manager.cc:1398] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.272816 30624 raft_consensus.cc:348] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.272909 30624 raft_consensus.cc:374] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.272939 30624 raft_consensus.cc:729] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.272998 30622 tablet_bootstrap.cc:492] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:16.273068 30624 consensus_queue.cc:260] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.273319 30624 ts_tablet_manager.cc:1398] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.273494 30624 tablet_bootstrap.cc:492] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:16.273522 30623 tablet_bootstrap.cc:654] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.273869 30622 tablet_bootstrap.cc:654] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.274446 30624 tablet_bootstrap.cc:654] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.274761 30623 tablet_bootstrap.cc:492] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:16.274832 30623 ts_tablet_manager.cc:1367] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:16.275163 30622 tablet_bootstrap.cc:492] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:16.275233 30622 ts_tablet_manager.cc:1367] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:16.275346 30623 raft_consensus.cc:348] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.275444 30623 raft_consensus.cc:374] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.275475 30623 raft_consensus.cc:729] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.275612 30623 consensus_queue.cc:260] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.275658 30622 raft_consensus.cc:348] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.275749 30622 raft_consensus.cc:374] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.275784 30622 raft_consensus.cc:729] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.275875 30623 ts_tablet_manager.cc:1398] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:16.275913 30622 consensus_queue.cc:260] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.276188 30622 ts_tablet_manager.cc:1398] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:16.276993 30627 tablet_bootstrap.cc:492] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:16.277086 30627 ts_tablet_manager.cc:1367] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:00:16.277560 30627 raft_consensus.cc:348] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.277673 30627 raft_consensus.cc:374] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.277704 30627 raft_consensus.cc:729] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.277819 30627 consensus_queue.cc:260] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.278126 30627 ts_tablet_manager.cc:1398] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:16.279685 30624 tablet_bootstrap.cc:492] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:16.279788 30624 ts_tablet_manager.cc:1367] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:16.280265 30624 raft_consensus.cc:348] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.280391 30624 raft_consensus.cc:374] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.280434 30624 raft_consensus.cc:729] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.280572 30624 consensus_queue.cc:260] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.280850 30624 ts_tablet_manager.cc:1398] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:16.280999 30624 tablet_bootstrap.cc:492] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:16.281927 30624 tablet_bootstrap.cc:654] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.283116 30624 tablet_bootstrap.cc:492] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:16.283205 30624 ts_tablet_manager.cc:1367] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:16.283586 30624 raft_consensus.cc:348] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.283676 30624 raft_consensus.cc:374] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.283699 30624 raft_consensus.cc:729] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.283790 30624 consensus_queue.cc:260] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.284035 30624 ts_tablet_manager.cc:1398] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:16.377602 30474 raft_consensus.cc:482] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:16.377738 30474 raft_consensus.cc:504] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.378237 30474 leader_election.cc:288] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:16.379137 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9618b854814e11a3c6ae7504821d35" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:16.379307 29733 raft_consensus.cc:2454] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:16.379652 29405 leader_election.cc:302] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:16.379833 30474 raft_consensus.cc:2781] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:16.379882 30474 raft_consensus.cc:482] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:16.379921 30474 raft_consensus.cc:3037] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.381214 30474 raft_consensus.cc:504] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.381583 30474 leader_election.cc:288] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:16.381948 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9618b854814e11a3c6ae7504821d35" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:16.382081 29733 raft_consensus.cc:3037] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.382081 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9618b854814e11a3c6ae7504821d35" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:16.382239 29864 raft_consensus.cc:2454] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:16.382458 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9618b854814e11a3c6ae7504821d35" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:16.382556 29863 raft_consensus.cc:3037] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.383085 29733 raft_consensus.cc:2454] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:16.383395 29405 leader_election.cc:302] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:16.383574 30474 raft_consensus.cc:2781] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:16.383654 30474 raft_consensus.cc:686] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:16.383796 30474 consensus_queue.cc:237] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.383985 29863 raft_consensus.cc:2454] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:16.385555 29326 catalog_manager.cc:5484] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:16.390116 30558 raft_consensus.cc:482] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:16.390214 30558 raft_consensus.cc:504] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.390625 30558 leader_election.cc:288] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:16.391068 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2db3a7fca4420a94891cd96c54ec3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:16.391192 29863 raft_consensus.cc:2454] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:16.391505 29536 leader_election.cc:302] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:16.391675 30558 raft_consensus.cc:2781] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:16.391737 30558 raft_consensus.cc:482] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:16.391769 30558 raft_consensus.cc:3037] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.392688 30558 raft_consensus.cc:504] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.392992 30558 leader_election.cc:288] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:16.393337 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2db3a7fca4420a94891cd96c54ec3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:16.393443 29863 raft_consensus.cc:3037] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.394337 29863 raft_consensus.cc:2454] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:16.394598 29536 leader_election.cc:302] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:16.394750 30558 raft_consensus.cc:2781] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:16.394825 30558 raft_consensus.cc:686] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:16.394986 30558 consensus_queue.cc:237] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:16.396251 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2db3a7fca4420a94891cd96c54ec3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:16.396271 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a2db3a7fca4420a94891cd96c54ec3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:16.396417 29471 raft_consensus.cc:2454] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:16.396754 29326 catalog_manager.cc:5484] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:16.603415 30558 raft_consensus.cc:482] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:16.603543 30558 raft_consensus.cc:504] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.603940 30558 leader_election.cc:288] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:16.604339 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cceaa60c7e470ba8b8d205d026625b" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:16.604410 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cceaa60c7e470ba8b8d205d026625b" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:16.604473 29863 raft_consensus.cc:2454] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:16.604570 29471 raft_consensus.cc:2454] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:16.604770 29536 leader_election.cc:302] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:16.604946 30558 raft_consensus.cc:2781] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:16.605005 30558 raft_consensus.cc:482] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:16.605031 30558 raft_consensus.cc:3037] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.605849 30558 raft_consensus.cc:504] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.606168 30558 leader_election.cc:288] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:16.606464 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cceaa60c7e470ba8b8d205d026625b" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:16.606506 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cceaa60c7e470ba8b8d205d026625b" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:16.606571 29863 raft_consensus.cc:3037] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.606643 29471 raft_consensus.cc:3037] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.607395 29863 raft_consensus.cc:2454] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:16.607442 29471 raft_consensus.cc:2454] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:16.607666 29536 leader_election.cc:302] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:16.607867 30558 raft_consensus.cc:2781] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:16.607949 30558 raft_consensus.cc:686] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:16.608078 30558 consensus_queue.cc:237] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.609642 29326 catalog_manager.cc:5484] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:00:16.745229 30640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:16.745533 30640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:16.748837 30640 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:00:16.804740 30474 consensus_queue.cc:1035] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.808972 30468 consensus_queue.cc:1035] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.841704 29864 raft_consensus.cc:1261] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.845575 29471 raft_consensus.cc:3037] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:16.845789 30558 consensus_queue.cc:1035] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.846346 29864 raft_consensus.cc:1261] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.846761 29471 raft_consensus.cc:1261] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.847127 30558 consensus_queue.cc:1035] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.847627 30558 consensus_queue.cc:1035] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:16.848009 29471 raft_consensus.cc:1261] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:16.848896 30558 consensus_queue.cc:1035] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.571 ms
time per row: 0.0547857 ms
I20240502 14:00:16.903029 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:16.908409 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc14b4bd42841df91972e0c5d02faa4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.908408 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc14b4bd42841df91972e0c5d02faa4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.908581 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc14b4bd42841df91972e0c5d02faa4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.908735 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc14b4bd42841df91972e0c5d02faa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.908735 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc14b4bd42841df91972e0c5d02faa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.908877 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc14b4bd42841df91972e0c5d02faa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.910606 30694 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:16.911255 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6ced4716f64a3aaad4cd5299e46d13 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.911499 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6ced4716f64a3aaad4cd5299e46d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.911496 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2ed5a4845b4bd78b19af6380c77982 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.911729 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2ed5a4845b4bd78b19af6380c77982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.911768 30695 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:16.911738 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6ced4716f64a3aaad4cd5299e46d13 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.911988 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6ced4716f64a3aaad4cd5299e46d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.912638 30695 tablet_bootstrap.cc:654] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.912647 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6ced4716f64a3aaad4cd5299e46d13 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.912878 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6ced4716f64a3aaad4cd5299e46d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.913854 30695 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:16.913944 30695 ts_tablet_manager.cc:1367] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:16.914361 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2ed5a4845b4bd78b19af6380c77982 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.914433 30695 raft_consensus.cc:348] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.914541 30695 raft_consensus.cc:374] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.914570 30695 raft_consensus.cc:729] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.914611 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2ed5a4845b4bd78b19af6380c77982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.914700 30695 consensus_queue.cc:260] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.915041 30695 ts_tablet_manager.cc:1398] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:16.914981 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2ed5a4845b4bd78b19af6380c77982 (DEFAULT_TABLE table=rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:16.915683 30695 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:16.915760 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2ed5a4845b4bd78b19af6380c77982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:16.917150 30695 tablet_bootstrap.cc:654] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.918419 30698 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:16.918594 30694 tablet_bootstrap.cc:654] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.919348 30698 tablet_bootstrap.cc:654] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.919790 30696 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:16.920549 30695 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:16.920610 30694 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:16.920645 30695 ts_tablet_manager.cc:1367] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:00:16.920686 30694 ts_tablet_manager.cc:1367] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:16.920704 30696 tablet_bootstrap.cc:654] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.921137 30695 raft_consensus.cc:348] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.921185 30694 raft_consensus.cc:348] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.921257 30695 raft_consensus.cc:374] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.921272 30694 raft_consensus.cc:374] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.921301 30695 raft_consensus.cc:729] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.921308 30694 raft_consensus.cc:729] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.921433 30695 consensus_queue.cc:260] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.921473 30694 consensus_queue.cc:260] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.921731 30695 ts_tablet_manager.cc:1398] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:16.921864 30698 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:16.921877 30694 ts_tablet_manager.cc:1398] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.921960 30698 ts_tablet_manager.cc:1367] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:16.922004 30694 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:16.922111 30696 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:16.922192 30696 ts_tablet_manager.cc:1367] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:16.922431 30698 raft_consensus.cc:348] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.922528 30698 raft_consensus.cc:374] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.922560 30698 raft_consensus.cc:729] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.922580 30696 raft_consensus.cc:348] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.922672 30696 raft_consensus.cc:374] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.922708 30696 raft_consensus.cc:729] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.922732 30694 tablet_bootstrap.cc:654] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.922688 30698 consensus_queue.cc:260] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.922821 30696 consensus_queue.cc:260] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.922983 30698 ts_tablet_manager.cc:1398] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:16.923100 30696 ts_tablet_manager.cc:1398] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.923143 30698 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:16.923341 30696 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:16.924068 30696 tablet_bootstrap.cc:654] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.924383 30698 tablet_bootstrap.cc:654] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.925029 30696 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:16.925107 30696 ts_tablet_manager.cc:1367] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:16.925513 30696 raft_consensus.cc:348] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.925611 30696 raft_consensus.cc:374] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.925644 30696 raft_consensus.cc:729] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.925753 30696 consensus_queue.cc:260] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.926035 30696 ts_tablet_manager.cc:1398] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.926203 30696 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:16.926733 30698 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:16.926815 30698 ts_tablet_manager.cc:1367] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:16.927068 30696 tablet_bootstrap.cc:654] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:16.927281 30698 raft_consensus.cc:348] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.927397 30698 raft_consensus.cc:374] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.927438 30698 raft_consensus.cc:729] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.927587 30698 consensus_queue.cc:260] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.927912 30698 ts_tablet_manager.cc:1398] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:16.928938 30696 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:16.929018 30696 ts_tablet_manager.cc:1367] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:16.929463 30696 raft_consensus.cc:348] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.929554 30696 raft_consensus.cc:374] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.929567 30694 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:16.929589 30696 raft_consensus.cc:729] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.929652 30694 ts_tablet_manager.cc:1367] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:16.929701 30696 consensus_queue.cc:260] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:16.929980 30696 ts_tablet_manager.cc:1398] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:16.930574 30694 raft_consensus.cc:348] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.930666 30694 raft_consensus.cc:374] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:16.930697 30694 raft_consensus.cc:729] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:16.930905 30694 consensus_queue.cc:260] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:16.931191 30694 ts_tablet_manager.cc:1398] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:17.059477 30598 raft_consensus.cc:482] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:17.059592 30598 raft_consensus.cc:504] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.060006 30598 leader_election.cc:288] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.060361 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6ced4716f64a3aaad4cd5299e46d13" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:17.060441 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6ced4716f64a3aaad4cd5299e46d13" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:17.060496 29733 raft_consensus.cc:2454] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:17.060592 29471 raft_consensus.cc:2454] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:17.060804 29798 leader_election.cc:302] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.061031 30598 raft_consensus.cc:2781] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:17.061095 30598 raft_consensus.cc:482] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:17.061126 30598 raft_consensus.cc:3037] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.061987 30598 raft_consensus.cc:504] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.062321 30598 leader_election.cc:288] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.062650 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6ced4716f64a3aaad4cd5299e46d13" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:17.062657 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6ced4716f64a3aaad4cd5299e46d13" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:17.062757 29733 raft_consensus.cc:3037] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.062767 29471 raft_consensus.cc:3037] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.063493 29733 raft_consensus.cc:2454] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:17.063585 29471 raft_consensus.cc:2454] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:17.063762 29798 leader_election.cc:302] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.063958 30598 raft_consensus.cc:2781] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:17.064172 30598 raft_consensus.cc:686] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:17.064313 30598 consensus_queue.cc:237] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.065905 29326 catalog_manager.cc:5484] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:17.154635 30466 raft_consensus.cc:482] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:17.154767 30466 raft_consensus.cc:504] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.155261 30466 leader_election.cc:288] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:17.155720 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2ed5a4845b4bd78b19af6380c77982" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:17.155889 29864 raft_consensus.cc:2454] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:17.156230 29667 leader_election.cc:302] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.156431 30466 raft_consensus.cc:2781] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:17.156491 30466 raft_consensus.cc:482] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:17.156545 30466 raft_consensus.cc:3037] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.157486 30466 raft_consensus.cc:504] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.157841 30466 leader_election.cc:288] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:17.158185 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2ed5a4845b4bd78b19af6380c77982" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:17.158325 29864 raft_consensus.cc:3037] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.159334 29864 raft_consensus.cc:2454] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:17.159350 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2ed5a4845b4bd78b19af6380c77982" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:17.159405 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2ed5a4845b4bd78b19af6380c77982" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:17.159520 29602 raft_consensus.cc:2454] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:17.159627 29667 leader_election.cc:302] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.159798 30466 raft_consensus.cc:2781] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:17.159988 30466 raft_consensus.cc:686] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:17.160137 30466 consensus_queue.cc:237] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.161775 29326 catalog_manager.cc:5484] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:17.202236 30466 raft_consensus.cc:482] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:17.202404 30466 raft_consensus.cc:504] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.202867 30466 leader_election.cc:288] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.203361 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:17.203377 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:17.203488 29601 raft_consensus.cc:2454] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:17.203536 29471 raft_consensus.cc:2454] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:17.203789 29666 leader_election.cc:302] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.203948 30466 raft_consensus.cc:2781] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:17.204003 30466 raft_consensus.cc:482] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:17.204030 30466 raft_consensus.cc:3037] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.204802 30466 raft_consensus.cc:504] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.205109 30466 leader_election.cc:288] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.205425 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:17.205462 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:17.205549 29601 raft_consensus.cc:3037] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.205564 29471 raft_consensus.cc:3037] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.206310 29601 raft_consensus.cc:2454] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:17.206512 29471 raft_consensus.cc:2454] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:17.206570 29666 leader_election.cc:302] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.206758 30466 raft_consensus.cc:2781] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:17.206908 30466 raft_consensus.cc:686] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:17.207098 30466 consensus_queue.cc:237] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.208704 29326 catalog_manager.cc:5484] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:00:17.327234 30713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:17.327481 30713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:17.330636 30713 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:00:17.406667 29864 raft_consensus.cc:1261] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:17.406764 29601 raft_consensus.cc:3037] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.407187 30710 consensus_queue.cc:1035] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:17.407927 29601 raft_consensus.cc:1261] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:17.409562 30466 consensus_queue.cc:1035] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:17.414714 29471 raft_consensus.cc:1261] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:17.414994 29733 raft_consensus.cc:1261] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:17.415355 30707 consensus_queue.cc:1035] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:17.416770 30707 consensus_queue.cc:1035] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:17.421355 29470 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:17.422258 30466 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:17.424511 29601 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:17.424908 30710 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 97.2118 ms
time per row: 0.0486059 ms
I20240502 14:00:17.472973 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:17.478547 29450 tablet_service.cc:1467] Processing CreateTablet for tablet ee26e51dcd054057ad08b6db2f4574d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.478570 29581 tablet_service.cc:1467] Processing CreateTablet for tablet ee26e51dcd054057ad08b6db2f4574d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.478919 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee26e51dcd054057ad08b6db2f4574d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.478919 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee26e51dcd054057ad08b6db2f4574d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.479359 29712 tablet_service.cc:1467] Processing CreateTablet for tablet ee26e51dcd054057ad08b6db2f4574d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.479454 29449 tablet_service.cc:1467] Processing CreateTablet for tablet d3ae9aa7809c4372aa9d322458baa6b8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.479617 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee26e51dcd054057ad08b6db2f4574d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.479672 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ae9aa7809c4372aa9d322458baa6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.481362 29451 tablet_service.cc:1467] Processing CreateTablet for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.481614 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.481853 30774 tablet_bootstrap.cc:492] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:17.482308 29843 tablet_service.cc:1467] Processing CreateTablet for tablet d3ae9aa7809c4372aa9d322458baa6b8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.482574 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ae9aa7809c4372aa9d322458baa6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.482690 30774 tablet_bootstrap.cc:654] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.484035 30773 tablet_bootstrap.cc:492] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:17.484861 30776 tablet_bootstrap.cc:492] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:17.485364 30773 tablet_bootstrap.cc:654] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.485673 29581 tablet_service.cc:1467] Processing CreateTablet for tablet d3ae9aa7809c4372aa9d322458baa6b8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.485848 30776 tablet_bootstrap.cc:654] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.485935 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3ae9aa7809c4372aa9d322458baa6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.486279 29712 tablet_service.cc:1467] Processing CreateTablet for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.486538 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.487003 30776 tablet_bootstrap.cc:492] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:17.487085 30776 ts_tablet_manager.cc:1367] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:17.487614 30776 raft_consensus.cc:348] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.487725 30776 raft_consensus.cc:374] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.487757 30776 raft_consensus.cc:729] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.487887 30776 consensus_queue.cc:260] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.488191 30776 ts_tablet_manager.cc:1398] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:17.488407 29843 tablet_service.cc:1467] Processing CreateTablet for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:17.488612 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.492419 30776 tablet_bootstrap.cc:492] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:17.492570 30772 tablet_bootstrap.cc:492] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:17.493412 30772 tablet_bootstrap.cc:654] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.493434 30776 tablet_bootstrap.cc:654] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.494544 30772 tablet_bootstrap.cc:492] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:17.494633 30772 ts_tablet_manager.cc:1367] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:17.494980 30776 tablet_bootstrap.cc:492] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:17.495110 30776 ts_tablet_manager.cc:1367] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:17.495169 30772 raft_consensus.cc:348] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.495337 30772 raft_consensus.cc:374] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.495388 30772 raft_consensus.cc:729] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.495555 30772 consensus_queue.cc:260] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.495669 30776 raft_consensus.cc:348] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.495826 30776 raft_consensus.cc:374] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.495890 30776 raft_consensus.cc:729] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.495906 30772 ts_tablet_manager.cc:1398] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:17.496062 30776 consensus_queue.cc:260] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.496105 30772 tablet_bootstrap.cc:492] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:17.497011 30772 tablet_bootstrap.cc:654] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.497596 30776 ts_tablet_manager.cc:1398] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:17.499686 30772 tablet_bootstrap.cc:492] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:17.499949 30772 ts_tablet_manager.cc:1367] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:17.500357 30772 raft_consensus.cc:348] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.500458 30772 raft_consensus.cc:374] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.500494 30772 raft_consensus.cc:729] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.500635 30772 consensus_queue.cc:260] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.500921 30772 ts_tablet_manager.cc:1398] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:17.501144 30772 tablet_bootstrap.cc:492] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:17.502753 30772 tablet_bootstrap.cc:654] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.504719 30772 tablet_bootstrap.cc:492] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:17.504809 30772 ts_tablet_manager.cc:1367] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:17.505180 30772 raft_consensus.cc:348] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.505275 30772 raft_consensus.cc:374] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.505308 30772 raft_consensus.cc:729] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.505445 30772 consensus_queue.cc:260] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.505733 30772 ts_tablet_manager.cc:1398] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:17.507720 30773 tablet_bootstrap.cc:492] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:17.507819 30773 ts_tablet_manager.cc:1367] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 14:00:17.507879 30774 tablet_bootstrap.cc:492] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:17.507989 30774 ts_tablet_manager.cc:1367] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240502 14:00:17.508270 30773 raft_consensus.cc:348] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.508364 30773 raft_consensus.cc:374] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.508400 30773 raft_consensus.cc:729] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.508482 30774 raft_consensus.cc:348] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.508515 30773 consensus_queue.cc:260] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.508570 30774 raft_consensus.cc:374] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.508591 30774 raft_consensus.cc:729] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.508698 30774 consensus_queue.cc:260] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.508769 30773 ts_tablet_manager.cc:1398] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:17.508953 30774 ts_tablet_manager.cc:1398] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:17.508936 30773 tablet_bootstrap.cc:492] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:17.509083 30774 tablet_bootstrap.cc:492] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:17.509934 30773 tablet_bootstrap.cc:654] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.510043 30774 tablet_bootstrap.cc:654] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:17.511209 30774 tablet_bootstrap.cc:492] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:17.511298 30774 ts_tablet_manager.cc:1367] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:17.511742 30774 raft_consensus.cc:348] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.511854 30774 raft_consensus.cc:374] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.511893 30774 raft_consensus.cc:729] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.512025 30774 consensus_queue.cc:260] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.512313 30774 ts_tablet_manager.cc:1398] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:17.512504 30773 tablet_bootstrap.cc:492] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:17.512606 30773 ts_tablet_manager.cc:1367] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:17.513025 30773 raft_consensus.cc:348] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.513131 30773 raft_consensus.cc:374] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:17.513159 30773 raft_consensus.cc:729] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:17.513284 30773 consensus_queue.cc:260] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.514746 30773 ts_tablet_manager.cc:1398] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:17.615101 30707 raft_consensus.cc:482] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:17.615242 30707 raft_consensus.cc:504] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.615655 30707 leader_election.cc:288] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.616079 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad03c8bc0c994bc6a9e17c1206f0d8b4" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:17.616111 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad03c8bc0c994bc6a9e17c1206f0d8b4" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:17.616204 29733 raft_consensus.cc:2454] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:17.616289 29470 raft_consensus.cc:2454] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:17.616541 29798 leader_election.cc:302] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.616694 30707 raft_consensus.cc:2781] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:17.616751 30707 raft_consensus.cc:482] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:17.616777 30707 raft_consensus.cc:3037] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.617550 30707 raft_consensus.cc:504] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.617842 30707 leader_election.cc:288] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.618163 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad03c8bc0c994bc6a9e17c1206f0d8b4" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:17.618264 29733 raft_consensus.cc:3037] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.619060 29733 raft_consensus.cc:2454] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:17.619268 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad03c8bc0c994bc6a9e17c1206f0d8b4" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:17.619321 29798 leader_election.cc:302] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:17.619374 29470 raft_consensus.cc:3037] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.619477 30707 raft_consensus.cc:2781] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:17.619578 30707 raft_consensus.cc:686] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:17.619719 30707 consensus_queue.cc:237] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.620255 29470 raft_consensus.cc:2454] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:17.621546 29323 catalog_manager.cc:5484] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:17.621687 30474 raft_consensus.cc:482] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:17.621764 30474 raft_consensus.cc:504] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.621778 30684 raft_consensus.cc:482] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:17.621877 30684 raft_consensus.cc:504] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.622119 30474 leader_election.cc:288] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:17.622254 30684 leader_election.cc:288] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:17.622462 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee26e51dcd054057ad08b6db2f4574d8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:17.622579 29601 raft_consensus.cc:2454] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:17.622787 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee26e51dcd054057ad08b6db2f4574d8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:17.622887 29404 leader_election.cc:302] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:17.622926 29732 raft_consensus.cc:2454] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:17.623052 30474 raft_consensus.cc:2781] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:17.623111 30474 raft_consensus.cc:482] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:17.623059 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee26e51dcd054057ad08b6db2f4574d8" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:17.623173 30474 raft_consensus.cc:3037] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.623980 30474 raft_consensus.cc:504] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.624199 29470 raft_consensus.cc:2379] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6e2e2c10def4331bec17bd110c9b423 in current term 1: Already voted for candidate 1d00d3a47c52478894cf8c66959ef7bd in this term.
I20240502 14:00:17.624322 30474 leader_election.cc:288] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:17.624644 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee26e51dcd054057ad08b6db2f4574d8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:17.624704 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee26e51dcd054057ad08b6db2f4574d8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:17.624752 29732 raft_consensus.cc:3037] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.624801 29601 raft_consensus.cc:3037] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.624807 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee26e51dcd054057ad08b6db2f4574d8" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:17.625679 29601 raft_consensus.cc:2454] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:17.625723 29732 raft_consensus.cc:2454] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:17.625923 29404 leader_election.cc:302] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:17.626086 30474 raft_consensus.cc:2781] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:17.626152 29537 leader_election.cc:302] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423; no voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:17.626282 30474 raft_consensus.cc:686] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:17.626415 30684 raft_consensus.cc:2726] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:00:17.626441 30474 consensus_queue.cc:237] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:17.628074 29323 catalog_manager.cc:5484] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:17.771476 30707 raft_consensus.cc:482] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:17.771618 30707 raft_consensus.cc:504] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.772034 30707 leader_election.cc:288] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.772485 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ae9aa7809c4372aa9d322458baa6b8" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:17.772521 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ae9aa7809c4372aa9d322458baa6b8" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:17.772629 29601 raft_consensus.cc:2454] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:17.772646 29470 raft_consensus.cc:2454] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:17.772931 29797 leader_election.cc:302] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:17.773136 30707 raft_consensus.cc:2781] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:17.773200 30707 raft_consensus.cc:482] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:17.773250 30707 raft_consensus.cc:3037] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.774258 30707 raft_consensus.cc:504] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.774655 30707 leader_election.cc:288] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:17.775063 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ae9aa7809c4372aa9d322458baa6b8" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:17.775081 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3ae9aa7809c4372aa9d322458baa6b8" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:17.775204 29601 raft_consensus.cc:3037] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.775204 29470 raft_consensus.cc:3037] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:17.776185 29470 raft_consensus.cc:2454] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:17.776185 29601 raft_consensus.cc:2454] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:17.776511 29797 leader_election.cc:302] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:17.776691 30707 raft_consensus.cc:2781] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:17.776777 30707 raft_consensus.cc:686] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:17.776917 30707 consensus_queue.cc:237] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:17.778789 29320 catalog_manager.cc:5484] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:00:17.961485 30789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:17.961768 30789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:17.964931 30789 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:00:18.025378 30474 consensus_queue.cc:1035] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.029158 30786 consensus_queue.cc:1035] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.033910 30707 consensus_queue.cc:1035] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.038296 30598 consensus_queue.cc:1035] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.058538 29601 raft_consensus.cc:1261] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.058538 29471 raft_consensus.cc:1261] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.059808 30707 consensus_queue.cc:1035] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.063367 30707 consensus_queue.cc:1035] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.167 ms
time per row: 0.0560835 ms
I20240502 14:00:18.120348 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:18.126127 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 925eac6300a44f17bbbf3fe742bc2a26 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.126341 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 463dddd3473a42a68b488e1c465e997f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.126439 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925eac6300a44f17bbbf3fe742bc2a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.126611 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dddd3473a42a68b488e1c465e997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.126801 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 925eac6300a44f17bbbf3fe742bc2a26 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.127105 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925eac6300a44f17bbbf3fe742bc2a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.129289 30852 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:18.130189 30852 tablet_bootstrap.cc:654] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.130457 29582 tablet_service.cc:1467] Processing CreateTablet for tablet b2c754e60c1d4f6f95a807e9644bd21f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.130705 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c754e60c1d4f6f95a807e9644bd21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.131459 30852 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:18.131552 30852 ts_tablet_manager.cc:1367] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:18.132041 30852 raft_consensus.cc:348] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.132160 30852 raft_consensus.cc:374] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.132205 30852 raft_consensus.cc:729] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.132340 30852 consensus_queue.cc:260] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.132740 30852 ts_tablet_manager.cc:1398] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.133174 29451 tablet_service.cc:1467] Processing CreateTablet for tablet b2c754e60c1d4f6f95a807e9644bd21f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.133464 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c754e60c1d4f6f95a807e9644bd21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.135401 29711 tablet_service.cc:1467] Processing CreateTablet for tablet b2c754e60c1d4f6f95a807e9644bd21f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.135464 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 925eac6300a44f17bbbf3fe742bc2a26 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.135707 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c754e60c1d4f6f95a807e9644bd21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.135783 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 463dddd3473a42a68b488e1c465e997f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.135898 30854 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:18.136162 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dddd3473a42a68b488e1c465e997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.136783 30854 tablet_bootstrap.cc:654] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.137598 30855 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:18.137751 30851 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:18.138265 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925eac6300a44f17bbbf3fe742bc2a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.138506 30855 tablet_bootstrap.cc:654] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.138511 30851 tablet_bootstrap.cc:654] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.139483 30855 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:18.139554 30855 ts_tablet_manager.cc:1367] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:18.139623 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 463dddd3473a42a68b488e1c465e997f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.139870 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dddd3473a42a68b488e1c465e997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.139933 30855 raft_consensus.cc:348] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.140022 30855 raft_consensus.cc:374] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.140049 30855 raft_consensus.cc:729] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.140159 30855 consensus_queue.cc:260] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.140412 30855 ts_tablet_manager.cc:1398] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.140556 30855 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:18.141278 30855 tablet_bootstrap.cc:654] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.142277 30855 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:18.142346 30855 ts_tablet_manager.cc:1367] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:18.142719 30855 raft_consensus.cc:348] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.142813 30855 raft_consensus.cc:374] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.142870 30855 raft_consensus.cc:729] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.142974 30855 consensus_queue.cc:260] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.143107 30852 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:18.143251 30855 ts_tablet_manager.cc:1398] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.143411 30855 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:18.143823 30851 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:18.143903 30851 ts_tablet_manager.cc:1367] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:18.144156 30852 tablet_bootstrap.cc:654] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.144225 30855 tablet_bootstrap.cc:654] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.144363 30851 raft_consensus.cc:348] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.144469 30851 raft_consensus.cc:374] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.144505 30851 raft_consensus.cc:729] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.144670 30851 consensus_queue.cc:260] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.145004 30851 ts_tablet_manager.cc:1398] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.145162 30851 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:18.145177 30852 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:18.145255 30852 ts_tablet_manager.cc:1367] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:18.145318 30855 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:18.145396 30855 ts_tablet_manager.cc:1367] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:18.145725 30852 raft_consensus.cc:348] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.145836 30852 raft_consensus.cc:374] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.145869 30852 raft_consensus.cc:729] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.145984 30852 consensus_queue.cc:260] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.146051 30851 tablet_bootstrap.cc:654] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.146126 30854 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:18.146118 30855 raft_consensus.cc:348] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.146210 30855 raft_consensus.cc:374] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.146229 30854 ts_tablet_manager.cc:1367] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:18.146246 30855 raft_consensus.cc:729] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.146255 30852 ts_tablet_manager.cc:1398] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.146347 30855 consensus_queue.cc:260] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.146637 30855 ts_tablet_manager.cc:1398] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:18.146682 30854 raft_consensus.cc:348] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.146790 30854 raft_consensus.cc:374] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.146821 30854 raft_consensus.cc:729] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.147280 30854 consensus_queue.cc:260] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.147578 30854 ts_tablet_manager.cc:1398] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.148120 30851 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:18.148197 30851 ts_tablet_manager.cc:1367] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:18.148634 30851 raft_consensus.cc:348] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.148720 30851 raft_consensus.cc:374] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.148754 30851 raft_consensus.cc:729] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.148867 30851 consensus_queue.cc:260] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.149122 30851 ts_tablet_manager.cc:1398] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.149284 30851 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:18.150229 30851 tablet_bootstrap.cc:654] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.151298 30851 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:18.151435 30851 ts_tablet_manager.cc:1367] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:18.151810 30851 raft_consensus.cc:348] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.151906 30851 raft_consensus.cc:374] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.151940 30851 raft_consensus.cc:729] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.152066 30851 consensus_queue.cc:260] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.152330 30851 ts_tablet_manager.cc:1398] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.202389 30817 raft_consensus.cc:482] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:18.202520 30817 raft_consensus.cc:504] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.202962 30817 leader_election.cc:288] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:18.203361 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dddd3473a42a68b488e1c465e997f" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:18.203361 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dddd3473a42a68b488e1c465e997f" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:18.203519 29864 raft_consensus.cc:2454] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:18.203519 29601 raft_consensus.cc:2454] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:18.203860 29667 leader_election.cc:302] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:18.204042 30817 raft_consensus.cc:2781] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:18.204116 30817 raft_consensus.cc:482] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:18.204162 30817 raft_consensus.cc:3037] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.205008 30817 raft_consensus.cc:504] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.205344 30817 leader_election.cc:288] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:18.205631 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dddd3473a42a68b488e1c465e997f" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:18.205658 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463dddd3473a42a68b488e1c465e997f" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:18.205753 29864 raft_consensus.cc:3037] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.205753 29601 raft_consensus.cc:3037] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.206710 29864 raft_consensus.cc:2454] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:18.206710 29601 raft_consensus.cc:2454] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:18.207012 29667 leader_election.cc:302] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:18.207195 30817 raft_consensus.cc:2781] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:18.207284 30817 raft_consensus.cc:686] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:18.207423 30817 consensus_queue.cc:237] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.208979 29325 catalog_manager.cc:5484] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:18.256331 30830 raft_consensus.cc:482] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:18.256472 30830 raft_consensus.cc:504] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.256886 30830 leader_election.cc:288] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:18.257345 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925eac6300a44f17bbbf3fe742bc2a26" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:18.257345 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925eac6300a44f17bbbf3fe742bc2a26" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:18.257491 29601 raft_consensus.cc:2454] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:18.257491 29732 raft_consensus.cc:2454] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:18.257793 29797 leader_election.cc:302] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.257956 30830 raft_consensus.cc:2781] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:18.258030 30830 raft_consensus.cc:482] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:18.258069 30830 raft_consensus.cc:3037] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.258828 30830 raft_consensus.cc:504] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.259138 30830 leader_election.cc:288] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:18.259464 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925eac6300a44f17bbbf3fe742bc2a26" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:18.259474 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925eac6300a44f17bbbf3fe742bc2a26" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:18.259570 29732 raft_consensus.cc:3037] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.259577 29601 raft_consensus.cc:3037] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.260268 29601 raft_consensus.cc:2454] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:18.260552 29797 leader_election.cc:302] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.260715 30830 raft_consensus.cc:2781] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:18.260795 30830 raft_consensus.cc:686] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:18.260936 30830 consensus_queue.cc:237] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.262617 29325 catalog_manager.cc:5484] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:18.275635 29732 raft_consensus.cc:2454] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:18.339349 30844 raft_consensus.cc:482] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:18.339475 30844 raft_consensus.cc:504] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.339897 30844 leader_election.cc:288] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:18.340210 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c754e60c1d4f6f95a807e9644bd21f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:18.340365 29601 raft_consensus.cc:2454] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:18.340320 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c754e60c1d4f6f95a807e9644bd21f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:18.340471 29732 raft_consensus.cc:2454] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:18.340637 29404 leader_election.cc:302] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.340787 30844 raft_consensus.cc:2781] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:18.340844 30844 raft_consensus.cc:482] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:18.340880 30844 raft_consensus.cc:3037] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.341747 30844 raft_consensus.cc:504] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.342103 30844 leader_election.cc:288] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:18.342341 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c754e60c1d4f6f95a807e9644bd21f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:18.342465 29601 raft_consensus.cc:3037] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.342480 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c754e60c1d4f6f95a807e9644bd21f" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:18.342581 29732 raft_consensus.cc:3037] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.343505 29601 raft_consensus.cc:2454] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:18.343505 29732 raft_consensus.cc:2454] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:18.343767 29404 leader_election.cc:302] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.343930 30844 raft_consensus.cc:2781] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:18.344003 30844 raft_consensus.cc:686] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:18.344146 30844 consensus_queue.cc:237] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.345952 29325 catalog_manager.cc:5484] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:18.483620 30867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:18.483880 30867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:18.487051 30867 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:00:18.563154 29732 raft_consensus.cc:1261] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.563542 30844 consensus_queue.cc:1035] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.565115 29602 raft_consensus.cc:1261] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.565908 29601 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.565922 29864 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.566233 29733 raft_consensus.cc:1261] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.567067 30707 consensus_queue.cc:1035] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.567390 30707 consensus_queue.cc:1035] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.567492 30817 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.568006 29602 raft_consensus.cc:1261] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.568421 30786 consensus_queue.cc:1035] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.568645 30817 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.752 ms
time per row: 0.0503761 ms
I20240502 14:00:18.630336 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:18.635725 29582 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf63852df44cf1b762b7a981428c97 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.636029 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf63852df44cf1b762b7a981428c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.636030 29451 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf63852df44cf1b762b7a981428c97 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.636246 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 59235c5f3daf4ff1afeab83efe018405 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.636338 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf63852df44cf1b762b7a981428c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.636381 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 74be4a1291264d3090849cb66bbca085 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.636464 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59235c5f3daf4ff1afeab83efe018405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.636603 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74be4a1291264d3090849cb66bbca085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.638273 30851 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:18.638587 30854 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:18.639115 30851 tablet_bootstrap.cc:654] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.639163 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 74be4a1291264d3090849cb66bbca085 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.639417 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74be4a1291264d3090849cb66bbca085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.639436 30854 tablet_bootstrap.cc:654] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.640283 30851 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:18.640367 30851 ts_tablet_manager.cc:1367] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:18.640496 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 74be4a1291264d3090849cb66bbca085 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.640657 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 59235c5f3daf4ff1afeab83efe018405 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.640749 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74be4a1291264d3090849cb66bbca085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.640878 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59235c5f3daf4ff1afeab83efe018405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.640802 30851 raft_consensus.cc:348] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.640918 30851 raft_consensus.cc:374] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.640949 30851 raft_consensus.cc:729] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.641060 30851 consensus_queue.cc:260] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.641362 30851 ts_tablet_manager.cc:1398] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.641470 30855 tablet_bootstrap.cc:492] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:18.641691 29843 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf63852df44cf1b762b7a981428c97 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.641932 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf63852df44cf1b762b7a981428c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.642371 30855 tablet_bootstrap.cc:654] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.643028 30852 tablet_bootstrap.cc:492] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:18.643072 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 59235c5f3daf4ff1afeab83efe018405 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:18.643287 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59235c5f3daf4ff1afeab83efe018405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.643854 30852 tablet_bootstrap.cc:654] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.644510 30855 tablet_bootstrap.cc:492] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:18.644587 30855 ts_tablet_manager.cc:1367] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:18.644992 30855 raft_consensus.cc:348] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.645087 30855 raft_consensus.cc:374] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.645131 30855 raft_consensus.cc:729] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.645246 30855 consensus_queue.cc:260] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.645525 30855 ts_tablet_manager.cc:1398] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.645668 30851 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:18.645972 30854 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:18.646063 30854 ts_tablet_manager.cc:1367] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:18.646533 30854 raft_consensus.cc:348] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.646647 30854 raft_consensus.cc:374] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.646679 30854 raft_consensus.cc:729] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.646798 30854 consensus_queue.cc:260] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.646908 30851 tablet_bootstrap.cc:654] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.647150 30854 ts_tablet_manager.cc:1398] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.647325 30854 tablet_bootstrap.cc:492] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:18.647933 30852 tablet_bootstrap.cc:492] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:18.648006 30852 ts_tablet_manager.cc:1367] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:18.648214 30854 tablet_bootstrap.cc:654] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.648479 30852 raft_consensus.cc:348] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.648579 30852 raft_consensus.cc:374] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.648607 30852 raft_consensus.cc:729] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.648725 30852 consensus_queue.cc:260] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.649065 30852 ts_tablet_manager.cc:1398] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.649107 30851 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:18.649184 30851 ts_tablet_manager.cc:1367] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:18.649247 30852 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:18.649571 30854 tablet_bootstrap.cc:492] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:18.649653 30854 ts_tablet_manager.cc:1367] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:18.649641 30851 raft_consensus.cc:348] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.649740 30851 raft_consensus.cc:374] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.649777 30851 raft_consensus.cc:729] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.649931 30851 consensus_queue.cc:260] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.650107 30852 tablet_bootstrap.cc:654] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.650060 30854 raft_consensus.cc:348] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.650168 30854 raft_consensus.cc:374] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.650204 30854 raft_consensus.cc:729] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.650329 30854 consensus_queue.cc:260] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.650583 30854 ts_tablet_manager.cc:1398] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.650740 30854 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:18.651623 30854 tablet_bootstrap.cc:654] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.651949 30852 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:18.652019 30852 ts_tablet_manager.cc:1367] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:18.652436 30852 raft_consensus.cc:348] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.652516 30852 raft_consensus.cc:374] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.652549 30852 raft_consensus.cc:729] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.652680 30854 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:18.652663 30852 consensus_queue.cc:260] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.652771 30854 ts_tablet_manager.cc:1367] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:18.652892 30852 ts_tablet_manager.cc:1398] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.653023 30852 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:18.653195 30854 raft_consensus.cc:348] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.653295 30854 raft_consensus.cc:374] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.653333 30854 raft_consensus.cc:729] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.653460 30854 consensus_queue.cc:260] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.653723 30854 ts_tablet_manager.cc:1398] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:18.653882 30852 tablet_bootstrap.cc:654] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:18.654228 30851 ts_tablet_manager.cc:1398] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:18.655119 30852 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:18.655210 30852 ts_tablet_manager.cc:1367] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:18.655622 30852 raft_consensus.cc:348] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.655707 30852 raft_consensus.cc:374] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:18.655737 30852 raft_consensus.cc:729] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:18.655853 30852 consensus_queue.cc:260] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.656108 30852 ts_tablet_manager.cc:1398] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:18.667698 30786 raft_consensus.cc:482] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:18.667816 30786 raft_consensus.cc:504] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.668226 30786 leader_election.cc:288] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:18.668573 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59235c5f3daf4ff1afeab83efe018405" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:18.668596 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59235c5f3daf4ff1afeab83efe018405" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:18.668706 29864 raft_consensus.cc:2454] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:18.668730 29602 raft_consensus.cc:2454] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:18.668987 29404 leader_election.cc:302] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.669149 30786 raft_consensus.cc:2781] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:18.669216 30786 raft_consensus.cc:482] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:18.669270 30786 raft_consensus.cc:3037] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.670219 30786 raft_consensus.cc:504] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.670514 30786 leader_election.cc:288] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:18.670879 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59235c5f3daf4ff1afeab83efe018405" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:18.670994 29602 raft_consensus.cc:3037] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.670980 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59235c5f3daf4ff1afeab83efe018405" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:18.671079 29864 raft_consensus.cc:3037] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.671725 29602 raft_consensus.cc:2454] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:18.671773 29864 raft_consensus.cc:2454] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:18.671996 29404 leader_election.cc:302] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.672168 30786 raft_consensus.cc:2781] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:18.672253 30786 raft_consensus.cc:686] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:18.672391 30786 consensus_queue.cc:237] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:18.673856 29325 catalog_manager.cc:5484] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:18.685921 30905 raft_consensus.cc:482] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:18.686026 30905 raft_consensus.cc:504] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.686409 30905 leader_election.cc:288] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:18.686754 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf63852df44cf1b762b7a981428c97" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:18.686764 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf63852df44cf1b762b7a981428c97" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:18.686902 29864 raft_consensus.cc:2454] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:18.686902 29470 raft_consensus.cc:2454] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:18.687222 29536 leader_election.cc:302] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.687388 30905 raft_consensus.cc:2781] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:18.687453 30905 raft_consensus.cc:482] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:18.687482 30905 raft_consensus.cc:3037] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.688360 30905 raft_consensus.cc:504] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.688661 30905 leader_election.cc:288] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:18.688927 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf63852df44cf1b762b7a981428c97" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:18.688930 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf63852df44cf1b762b7a981428c97" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:18.689046 29470 raft_consensus.cc:3037] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.689046 29864 raft_consensus.cc:3037] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.689941 29864 raft_consensus.cc:2454] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:18.689941 29470 raft_consensus.cc:2454] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:18.690208 29537 leader_election.cc:302] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:18.690368 30905 raft_consensus.cc:2781] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:18.690447 30905 raft_consensus.cc:686] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:18.690601 30905 consensus_queue.cc:237] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:18.692124 29325 catalog_manager.cc:5484] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:18.761298 30786 raft_consensus.cc:482] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:18.761438 30786 raft_consensus.cc:504] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.761871 30786 leader_election.cc:288] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:18.762373 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74be4a1291264d3090849cb66bbca085" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:18.762373 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74be4a1291264d3090849cb66bbca085" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:18.762521 29864 raft_consensus.cc:2454] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:18.762521 29733 raft_consensus.cc:2454] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:18.762832 29405 leader_election.cc:302] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:18.763022 30786 raft_consensus.cc:2781] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:18.763103 30786 raft_consensus.cc:482] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:18.763152 30786 raft_consensus.cc:3037] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.764163 30786 raft_consensus.cc:504] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.764524 30786 leader_election.cc:288] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:18.764848 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74be4a1291264d3090849cb66bbca085" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:18.764969 29864 raft_consensus.cc:3037] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.764938 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74be4a1291264d3090849cb66bbca085" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:18.765082 29733 raft_consensus.cc:3037] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:18.765749 29864 raft_consensus.cc:2454] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:18.765890 29733 raft_consensus.cc:2454] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:18.766008 29405 leader_election.cc:302] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:18.766175 30786 raft_consensus.cc:2781] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:18.766263 30786 raft_consensus.cc:686] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:18.766422 30786 consensus_queue.cc:237] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:18.768074 29325 catalog_manager.cc:5484] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:18.902062 30924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:18.902338 30924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:18.905424 30924 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 14:00:18.983990 29733 raft_consensus.cc:1261] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.984472 29864 raft_consensus.cc:1261] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.985575 30474 consensus_queue.cc:1035] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.985618 30844 consensus_queue.cc:1035] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.985579 29470 raft_consensus.cc:1261] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.986302 29864 raft_consensus.cc:1261] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:18.986477 30905 consensus_queue.cc:1035] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.987095 30905 consensus_queue.cc:1035] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:18.997341 30814 mvcc.cc:205] Tried to move back new op lower bound from 7023240884160413696 to 7023240882957971456. Current Snapshot: MvccSnapshot[applied={T|T < 7023240884160413696}]
I20240502 14:00:18.999037 29864 raft_consensus.cc:1261] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.003700 29602 raft_consensus.cc:1261] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.003728 30844 consensus_queue.cc:1035] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.004169 30837 mvcc.cc:205] Tried to move back new op lower bound from 7023240884153942016 to 7023240883268599808. Current Snapshot: MvccSnapshot[applied={T|T < 7023240884153942016}]
I20240502 14:00:19.004313 30786 consensus_queue.cc:1035] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.010066 30816 mvcc.cc:205] Tried to move back new op lower bound from 7023240884153942016 to 7023240883268599808. Current Snapshot: MvccSnapshot[applied={T|T < 7023240884153942016}]
I20240502 14:00:19.017614 30829 mvcc.cc:205] Tried to move back new op lower bound from 7023240884160413696 to 7023240882957971456. Current Snapshot: MvccSnapshot[applied={T|T < 7023240884160413696}]
INSERT report
rows total: 2000
time total: 106.629 ms
time per row: 0.0533145 ms
I20240502 14:00:19.054750 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:19.060127 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae206e12b44b809d8b2fcc21611a28 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.060341 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae206e12b44b809d8b2fcc21611a28 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.060456 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae206e12b44b809d8b2fcc21611a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.060477 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae206e12b44b809d8b2fcc21611a28 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.060618 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae206e12b44b809d8b2fcc21611a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.060737 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae206e12b44b809d8b2fcc21611a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.061165 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8885e2c89645d78b65b8a486ecd06b (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.061375 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8885e2c89645d78b65b8a486ecd06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.062768 30851 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:19.062768 30854 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:19.062940 30855 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.063658 30854 tablet_bootstrap.cc:654] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.063658 30851 tablet_bootstrap.cc:654] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.063841 30855 tablet_bootstrap.cc:654] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.064509 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1a1a1592e84b3898a06d4360bac846 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.064512 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8885e2c89645d78b65b8a486ecd06b (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.064832 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1a1a1592e84b3898a06d4360bac846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.065198 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8885e2c89645d78b65b8a486ecd06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.065290 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8885e2c89645d78b65b8a486ecd06b (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.065513 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8885e2c89645d78b65b8a486ecd06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.067077 30852 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:19.065307 30855 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.067682 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1a1a1592e84b3898a06d4360bac846 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.067791 30855 ts_tablet_manager.cc:1367] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:19.067921 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1a1a1592e84b3898a06d4360bac846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.067961 30852 tablet_bootstrap.cc:654] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.068336 30855 raft_consensus.cc:348] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.068452 30855 raft_consensus.cc:374] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.068483 30855 raft_consensus.cc:729] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.068600 30855 consensus_queue.cc:260] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.068918 30855 ts_tablet_manager.cc:1398] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.069092 30855 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.069164 30852 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:19.069258 30852 ts_tablet_manager.cc:1367] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:19.069686 30852 raft_consensus.cc:348] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.069787 30852 raft_consensus.cc:374] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.069854 30852 raft_consensus.cc:729] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.069936 30855 tablet_bootstrap.cc:654] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.069998 30852 consensus_queue.cc:260] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.070313 30852 ts_tablet_manager.cc:1398] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.070501 30851 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:19.070557 30852 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:19.070617 30851 ts_tablet_manager.cc:1367] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:00:19.071158 30851 raft_consensus.cc:348] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.071269 30851 raft_consensus.cc:374] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.071302 30851 raft_consensus.cc:729] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.071431 30852 tablet_bootstrap.cc:654] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.071425 30851 consensus_queue.cc:260] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.071708 30851 ts_tablet_manager.cc:1398] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:19.071862 30851 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:19.072374 30852 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:19.072458 30852 ts_tablet_manager.cc:1367] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:19.072870 30852 raft_consensus.cc:348] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.072957 30852 raft_consensus.cc:374] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.072994 30851 tablet_bootstrap.cc:654] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.072990 30852 raft_consensus.cc:729] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.073127 30852 consensus_queue.cc:260] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.073354 30852 ts_tablet_manager.cc:1398] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:19.073688 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1a1a1592e84b3898a06d4360bac846 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.073904 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1a1a1592e84b3898a06d4360bac846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.075210 30851 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:19.075305 30851 ts_tablet_manager.cc:1367] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:19.075747 30851 raft_consensus.cc:348] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.075838 30851 raft_consensus.cc:374] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.075877 30851 raft_consensus.cc:729] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.075980 30851 consensus_queue.cc:260] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.076220 30851 ts_tablet_manager.cc:1398] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.077988 30855 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.078076 30855 ts_tablet_manager.cc:1367] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:19.078473 30855 raft_consensus.cc:348] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.078558 30855 raft_consensus.cc:374] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.078591 30855 raft_consensus.cc:729] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.078714 30855 consensus_queue.cc:260] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.078985 30855 ts_tablet_manager.cc:1398] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.079190 30855 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.079716 30854 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:19.079810 30854 ts_tablet_manager.cc:1367] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:19.080096 30855 tablet_bootstrap.cc:654] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.080333 30854 raft_consensus.cc:348] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.080451 30854 raft_consensus.cc:374] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.080484 30854 raft_consensus.cc:729] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.080610 30854 consensus_queue.cc:260] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.080895 30854 ts_tablet_manager.cc:1398] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.081179 30855 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.081204 30854 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:19.081266 30855 ts_tablet_manager.cc:1367] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:19.081722 30855 raft_consensus.cc:348] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.081789 30855 raft_consensus.cc:374] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.081807 30855 raft_consensus.cc:729] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.081871 30855 consensus_queue.cc:260] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.082053 30854 tablet_bootstrap.cc:654] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.082078 30855 ts_tablet_manager.cc:1398] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.083709 30854 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:19.083822 30854 ts_tablet_manager.cc:1367] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:19.084261 30854 raft_consensus.cc:348] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.084400 30854 raft_consensus.cc:374] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.084439 30854 raft_consensus.cc:729] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.084570 30854 consensus_queue.cc:260] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.084839 30854 ts_tablet_manager.cc:1398] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.085317 30905 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.085412 30905 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.085739 30905 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:19.086033 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae206e12b44b809d8b2fcc21611a28" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:19.086117 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae206e12b44b809d8b2fcc21611a28" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:19.086150 29733 raft_consensus.cc:2454] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:19.086226 29470 raft_consensus.cc:2454] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:19.086426 29536 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:19.086597 30905 raft_consensus.cc:2781] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.086665 30905 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.086704 30905 raft_consensus.cc:3037] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.087594 30905 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.087924 30905 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:19.088192 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae206e12b44b809d8b2fcc21611a28" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:19.088336 29733 raft_consensus.cc:3037] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.088354 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae206e12b44b809d8b2fcc21611a28" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:19.088443 29470 raft_consensus.cc:3037] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.089228 29733 raft_consensus.cc:2454] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:19.089351 29470 raft_consensus.cc:2454] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:19.089596 29536 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:19.089803 30905 raft_consensus.cc:2781] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.089912 30905 raft_consensus.cc:686] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:19.090047 30905 consensus_queue.cc:237] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.091455 29327 catalog_manager.cc:5484] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:19.143309 30844 raft_consensus.cc:482] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.143456 30844 raft_consensus.cc:504] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.143895 30844 leader_election.cc:288] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.144273 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1a1a1592e84b3898a06d4360bac846" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:19.144340 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1a1a1592e84b3898a06d4360bac846" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:19.144418 29864 raft_consensus.cc:2454] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:19.144507 29733 raft_consensus.cc:2454] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:19.144727 29405 leader_election.cc:302] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:19.144897 30786 raft_consensus.cc:2781] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.144954 30786 raft_consensus.cc:482] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.145006 30786 raft_consensus.cc:3037] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.145825 30786 raft_consensus.cc:504] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.146080 30786 leader_election.cc:288] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.146414 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1a1a1592e84b3898a06d4360bac846" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:19.146485 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1a1a1592e84b3898a06d4360bac846" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:19.146531 29864 raft_consensus.cc:3037] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.146598 29733 raft_consensus.cc:3037] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.147328 29864 raft_consensus.cc:2454] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:19.147421 29733 raft_consensus.cc:2454] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:19.147647 29405 leader_election.cc:302] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:19.147821 30786 raft_consensus.cc:2781] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.147900 30786 raft_consensus.cc:686] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:19.148041 30786 consensus_queue.cc:237] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.149600 29327 catalog_manager.cc:5484] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:19.174361 30707 raft_consensus.cc:482] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.174538 30707 raft_consensus.cc:504] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.175120 30707 leader_election.cc:288] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:19.175565 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8885e2c89645d78b65b8a486ecd06b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:19.175702 29733 raft_consensus.cc:2454] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:19.175760 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8885e2c89645d78b65b8a486ecd06b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:19.175889 29601 raft_consensus.cc:2454] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:19.176007 29798 leader_election.cc:302] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:19.176210 30707 raft_consensus.cc:2781] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.176268 30707 raft_consensus.cc:482] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.176317 30707 raft_consensus.cc:3037] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.177214 30707 raft_consensus.cc:504] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.177578 30707 leader_election.cc:288] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:19.178018 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8885e2c89645d78b65b8a486ecd06b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:19.178169 29601 raft_consensus.cc:3037] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.178629 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8885e2c89645d78b65b8a486ecd06b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:19.178755 29733 raft_consensus.cc:3037] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.179201 29601 raft_consensus.cc:2454] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.179503 29797 leader_election.cc:302] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:19.179642 30707 raft_consensus.cc:2781] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.179749 30707 raft_consensus.cc:686] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:19.179800 29733 raft_consensus.cc:2454] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.179878 30707 consensus_queue.cc:237] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:19.181558 29327 catalog_manager.cc:5484] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
W20240502 14:00:19.297209 30975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:19.297492 30975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:19.300767 30975 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 14:00:19.390735 29470 raft_consensus.cc:1261] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.391372 29864 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.391970 30905 consensus_queue.cc:1035] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.392935 29733 raft_consensus.cc:1261] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.393581 30684 consensus_queue.cc:1035] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.393932 30786 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.394412 29733 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.395639 29733 raft_consensus.cc:1261] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.395689 30786 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.396441 29600 raft_consensus.cc:1261] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.404274 30830 consensus_queue.cc:1035] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.404739 30830 consensus_queue.cc:1035] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.6745 ms
time per row: 0.0498372 ms
I20240502 14:00:19.448042 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:19.454412 29450 tablet_service.cc:1467] Processing CreateTablet for tablet f9a7fdccb29d49499abaf50d0725153d (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.454473 29581 tablet_service.cc:1467] Processing CreateTablet for tablet f9a7fdccb29d49499abaf50d0725153d (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.454725 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a7fdccb29d49499abaf50d0725153d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.454676 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 36daa6b40d8a48b8956b67a97833eff5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.454764 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a7fdccb29d49499abaf50d0725153d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.454913 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36daa6b40d8a48b8956b67a97833eff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.456578 30851 tablet_bootstrap.cc:492] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:19.456827 30854 tablet_bootstrap.cc:492] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:19.457473 30851 tablet_bootstrap.cc:654] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.457775 30854 tablet_bootstrap.cc:654] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.458668 30851 tablet_bootstrap.cc:492] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:19.458757 30851 ts_tablet_manager.cc:1367] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:19.459280 29712 tablet_service.cc:1467] Processing CreateTablet for tablet f9a7fdccb29d49499abaf50d0725153d (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.459373 30851 raft_consensus.cc:348] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.459470 30851 raft_consensus.cc:374] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.459504 30851 raft_consensus.cc:729] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.459565 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a7fdccb29d49499abaf50d0725153d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.459626 30851 consensus_queue.cc:260] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.459897 30851 ts_tablet_manager.cc:1398] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.460067 30851 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:19.460883 30851 tablet_bootstrap.cc:654] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.461678 30855 tablet_bootstrap.cc:492] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.461865 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebf61381a92430caa0c38cc04d14cb6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.462090 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ebf61381a92430caa0c38cc04d14cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.462334 30855 tablet_bootstrap.cc:654] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.463241 30851 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:19.463316 30851 ts_tablet_manager.cc:1367] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:19.463639 30851 raft_consensus.cc:348] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.463709 30851 raft_consensus.cc:374] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.463729 30851 raft_consensus.cc:729] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.463825 30851 consensus_queue.cc:260] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.464057 30851 ts_tablet_manager.cc:1398] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.464423 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 36daa6b40d8a48b8956b67a97833eff5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.464579 30855 tablet_bootstrap.cc:492] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.464648 30855 ts_tablet_manager.cc:1367] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:19.464423 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebf61381a92430caa0c38cc04d14cb6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.464695 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36daa6b40d8a48b8956b67a97833eff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.464803 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ebf61381a92430caa0c38cc04d14cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.465067 30855 raft_consensus.cc:348] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.465166 30855 raft_consensus.cc:374] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.465210 30855 raft_consensus.cc:729] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.465317 30855 consensus_queue.cc:260] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.465632 30855 ts_tablet_manager.cc:1398] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.465793 30855 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.466481 30855 tablet_bootstrap.cc:654] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.467895 30855 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.468004 30855 ts_tablet_manager.cc:1367] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:19.468039 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 7ebf61381a92430caa0c38cc04d14cb6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.468039 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 36daa6b40d8a48b8956b67a97833eff5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.468379 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ebf61381a92430caa0c38cc04d14cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.468611 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36daa6b40d8a48b8956b67a97833eff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.468472 30855 raft_consensus.cc:348] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.468748 30855 raft_consensus.cc:374] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.468796 30855 raft_consensus.cc:729] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.468927 30855 consensus_queue.cc:260] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.469237 30855 ts_tablet_manager.cc:1398] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.470229 30852 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:19.470932 30852 tablet_bootstrap.cc:654] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.470950 30854 tablet_bootstrap.cc:492] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:19.471031 30854 ts_tablet_manager.cc:1367] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:19.471448 30854 raft_consensus.cc:348] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.471575 30854 raft_consensus.cc:374] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.471616 30854 raft_consensus.cc:729] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.471767 30854 consensus_queue.cc:260] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.472128 30852 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:19.472221 30852 ts_tablet_manager.cc:1367] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:19.472659 30852 raft_consensus.cc:348] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.472764 30852 raft_consensus.cc:374] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.472798 30852 raft_consensus.cc:729] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.472919 30852 consensus_queue.cc:260] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.473182 30852 ts_tablet_manager.cc:1398] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:19.473387 30854 ts_tablet_manager.cc:1398] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:19.473556 30854 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:19.473564 30852 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:19.474092 30830 raft_consensus.cc:482] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.474175 30830 raft_consensus.cc:504] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.474229 30852 tablet_bootstrap.cc:654] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.474354 30854 tablet_bootstrap.cc:654] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.474581 30830 leader_election.cc:288] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.474877 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebf61381a92430caa0c38cc04d14cb6" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:19.475287 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebf61381a92430caa0c38cc04d14cb6" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:19.475404 29733 raft_consensus.cc:2454] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
W20240502 14:00:19.475430 29799 leader_election.cc:341] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:19.475512 30852 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:19.475574 30852 ts_tablet_manager.cc:1367] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:19.475886 30852 raft_consensus.cc:348] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.475960 30852 raft_consensus.cc:374] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.475991 30852 raft_consensus.cc:729] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.476094 30852 consensus_queue.cc:260] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.476346 30852 ts_tablet_manager.cc:1398] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:19.476444 29798 leader_election.cc:302] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:19.476619 30830 raft_consensus.cc:2781] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.476683 30830 raft_consensus.cc:482] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.476722 30830 raft_consensus.cc:3037] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.477192 30854 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:19.477260 30854 ts_tablet_manager.cc:1367] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:19.477640 30854 raft_consensus.cc:348] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.477723 30854 raft_consensus.cc:374] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.477757 30854 raft_consensus.cc:729] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.477761 30830 raft_consensus.cc:504] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.477887 30854 consensus_queue.cc:260] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.478108 30830 leader_election.cc:288] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.478202 30854 ts_tablet_manager.cc:1398] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.478367 30854 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:19.479287 30854 tablet_bootstrap.cc:654] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.480055 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebf61381a92430caa0c38cc04d14cb6" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:19.480175 29471 raft_consensus.cc:3037] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.480861 29471 raft_consensus.cc:2454] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.481107 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ebf61381a92430caa0c38cc04d14cb6" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:19.481155 29799 leader_election.cc:302] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:19.481216 29733 raft_consensus.cc:3037] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.481377 30830 raft_consensus.cc:2781] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.481487 30830 raft_consensus.cc:686] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:19.481591 30830 consensus_queue.cc:237] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.482230 29733 raft_consensus.cc:2454] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.482800 30854 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:19.482900 30854 ts_tablet_manager.cc:1367] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:00:19.483337 30854 raft_consensus.cc:348] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.483453 30854 raft_consensus.cc:374] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.483489 30854 raft_consensus.cc:729] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.483465 29323 catalog_manager.cc:5484] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:19.483601 30854 consensus_queue.cc:260] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.483842 30854 ts_tablet_manager.cc:1398] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:19.485419 30684 raft_consensus.cc:482] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.485524 30684 raft_consensus.cc:504] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.485934 30684 leader_election.cc:288] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.486331 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a7fdccb29d49499abaf50d0725153d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:19.486443 29471 raft_consensus.cc:2454] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:19.486454 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a7fdccb29d49499abaf50d0725153d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:19.486543 29733 raft_consensus.cc:2454] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:19.486747 29537 leader_election.cc:302] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:19.487000 30684 raft_consensus.cc:2781] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.487071 30684 raft_consensus.cc:482] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.487125 30684 raft_consensus.cc:3037] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.488024 30684 raft_consensus.cc:504] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.488380 30684 leader_election.cc:288] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.488781 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a7fdccb29d49499abaf50d0725153d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:19.488880 29733 raft_consensus.cc:3037] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.489670 29733 raft_consensus.cc:2454] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:19.489962 29536 leader_election.cc:302] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:19.490044 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a7fdccb29d49499abaf50d0725153d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:19.490123 30684 raft_consensus.cc:2781] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.490159 29471 raft_consensus.cc:3037] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.490211 30684 raft_consensus.cc:686] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:19.490360 30684 consensus_queue.cc:237] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.490985 29471 raft_consensus.cc:2454] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:19.492089 29326 catalog_manager.cc:5484] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:19.560580 30830 raft_consensus.cc:482] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.560707 30830 raft_consensus.cc:504] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.561116 30830 leader_election.cc:288] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:19.561416 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36daa6b40d8a48b8956b67a97833eff5" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:19.561480 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36daa6b40d8a48b8956b67a97833eff5" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:19.561558 29600 raft_consensus.cc:2454] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:19.561611 29471 raft_consensus.cc:2454] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:19.561822 29797 leader_election.cc:302] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:19.562021 30830 raft_consensus.cc:2781] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.562074 30830 raft_consensus.cc:482] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.562099 30830 raft_consensus.cc:3037] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.562908 30830 raft_consensus.cc:504] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.563216 30830 leader_election.cc:288] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:19.563720 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36daa6b40d8a48b8956b67a97833eff5" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:19.563824 29600 raft_consensus.cc:3037] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.563926 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36daa6b40d8a48b8956b67a97833eff5" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:19.564036 29471 raft_consensus.cc:3037] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.564868 29600 raft_consensus.cc:2454] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.564939 29471 raft_consensus.cc:2454] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.565191 29797 leader_election.cc:302] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:19.565383 30830 raft_consensus.cc:2781] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.565470 30830 raft_consensus.cc:686] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:19.565611 30830 consensus_queue.cc:237] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:19.567309 29326 catalog_manager.cc:5484] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:00:19.690768 31027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:19.691045 31027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:19.694180 31027 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 14:00:19.792480 29733 raft_consensus.cc:1261] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.792685 29729 raft_consensus.cc:1261] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.792934 30905 consensus_queue.cc:1035] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.793457 30830 consensus_queue.cc:1035] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.794483 29471 raft_consensus.cc:1261] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.799351 30905 consensus_queue.cc:1035] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.804024 29470 raft_consensus.cc:1261] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.806450 29471 raft_consensus.cc:1261] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.807570 30830 consensus_queue.cc:1035] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.807658 29600 raft_consensus.cc:1261] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:19.807852 30830 consensus_queue.cc:1035] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:19.808687 30707 consensus_queue.cc:1035] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.101 ms
time per row: 0.0615506 ms
I20240502 14:00:19.860054 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:19.865412 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 0867c6a22c2c4efd9a9babff371cc8ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.865741 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0867c6a22c2c4efd9a9babff371cc8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.865698 29581 tablet_service.cc:1467] Processing CreateTablet for tablet f3517a9ff83644ccb5cd4c0ebbd6e929 (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.865944 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3517a9ff83644ccb5cd4c0ebbd6e929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.866042 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 0867c6a22c2c4efd9a9babff371cc8ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.866276 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0867c6a22c2c4efd9a9babff371cc8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.867367 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 0867c6a22c2c4efd9a9babff371cc8ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.867367 29842 tablet_service.cc:1467] Processing CreateTablet for tablet f3517a9ff83644ccb5cd4c0ebbd6e929 (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.867697 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0867c6a22c2c4efd9a9babff371cc8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.867803 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3517a9ff83644ccb5cd4c0ebbd6e929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.868870 30851 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:19.868939 29713 tablet_service.cc:1467] Processing CreateTablet for tablet f3517a9ff83644ccb5cd4c0ebbd6e929 (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.869194 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3517a9ff83644ccb5cd4c0ebbd6e929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.869798 30852 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:19.869896 30855 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.869977 30851 tablet_bootstrap.cc:654] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.868939 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 253ea421888249d180928ba0aef28ddd (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.870170 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253ea421888249d180928ba0aef28ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.870759 30852 tablet_bootstrap.cc:654] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.870784 30855 tablet_bootstrap.cc:654] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.871160 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 253ea421888249d180928ba0aef28ddd (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.871436 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253ea421888249d180928ba0aef28ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.873490 30854 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:19.873665 30852 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:19.873747 30852 ts_tablet_manager.cc:1367] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:00:19.874191 30852 raft_consensus.cc:348] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.874305 30852 raft_consensus.cc:374] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.874343 30852 raft_consensus.cc:729] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.874378 30854 tablet_bootstrap.cc:654] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.874485 30852 consensus_queue.cc:260] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.874802 30852 ts_tablet_manager.cc:1398] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.875010 30852 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:19.875802 30852 tablet_bootstrap.cc:654] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.876441 30855 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.876554 30855 ts_tablet_manager.cc:1367] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:19.877009 30855 raft_consensus.cc:348] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.877120 30855 raft_consensus.cc:374] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.877151 30855 raft_consensus.cc:729] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.877269 30855 consensus_queue.cc:260] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.877379 30852 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:19.877458 30852 ts_tablet_manager.cc:1367] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:19.877555 30855 ts_tablet_manager.cc:1398] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.877714 30855 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.877813 30852 raft_consensus.cc:348] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.877894 30852 raft_consensus.cc:374] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.877915 30852 raft_consensus.cc:729] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.878037 30852 consensus_queue.cc:260] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.878317 30852 ts_tablet_manager.cc:1398] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:19.878789 30855 tablet_bootstrap.cc:654] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.879077 30854 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:19.879175 30854 ts_tablet_manager.cc:1367] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:00:19.879791 30854 raft_consensus.cc:348] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.879906 30854 raft_consensus.cc:374] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.879940 30854 raft_consensus.cc:729] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.880065 30854 consensus_queue.cc:260] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.880072 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 253ea421888249d180928ba0aef28ddd (DEFAULT_TABLE table=rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:19.880399 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253ea421888249d180928ba0aef28ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:19.880928 30854 ts_tablet_manager.cc:1398] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:19.884187 30855 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.884270 30855 ts_tablet_manager.cc:1367] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:19.884608 30855 raft_consensus.cc:348] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.884694 30855 raft_consensus.cc:374] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.884727 30855 raft_consensus.cc:729] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.884825 30855 consensus_queue.cc:260] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.885072 30855 ts_tablet_manager.cc:1398] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:19.885233 30855 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:19.886140 30855 tablet_bootstrap.cc:654] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.887049 30851 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:19.887158 30851 ts_tablet_manager.cc:1367] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:00:19.887590 30851 raft_consensus.cc:348] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.887686 30851 raft_consensus.cc:374] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.887719 30851 raft_consensus.cc:729] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.887842 30851 consensus_queue.cc:260] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.888101 30851 ts_tablet_manager.cc:1398] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.888267 30851 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:19.889001 30851 tablet_bootstrap.cc:654] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.890009 30851 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:19.890098 30851 ts_tablet_manager.cc:1367] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:19.890453 30855 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:19.890545 30855 ts_tablet_manager.cc:1367] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:19.890539 30851 raft_consensus.cc:348] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.890646 30851 raft_consensus.cc:374] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.890694 30851 raft_consensus.cc:729] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.891041 30855 raft_consensus.cc:348] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.891146 30855 raft_consensus.cc:374] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.891184 30855 raft_consensus.cc:729] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.891309 30855 consensus_queue.cc:260] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.891592 30855 ts_tablet_manager.cc:1398] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.891405 30851 consensus_queue.cc:260] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.892241 30851 ts_tablet_manager.cc:1398] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:19.892400 30851 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:19.893157 30851 tablet_bootstrap.cc:654] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:19.894246 30851 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:19.894325 30851 ts_tablet_manager.cc:1367] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:19.894680 30851 raft_consensus.cc:348] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.894765 30851 raft_consensus.cc:374] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:19.894805 30851 raft_consensus.cc:729] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:19.894966 30851 consensus_queue.cc:260] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:19.895310 30851 ts_tablet_manager.cc:1398] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.901268 30905 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.901381 30905 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.901731 30905 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:19.902065 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0867c6a22c2c4efd9a9babff371cc8ba" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:19.902161 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0867c6a22c2c4efd9a9babff371cc8ba" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:19.902235 29729 raft_consensus.cc:2454] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:19.902297 29862 raft_consensus.cc:2454] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:19.902529 29536 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:19.902700 30905 raft_consensus.cc:2781] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.902760 30905 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.902788 30905 raft_consensus.cc:3037] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.903592 30905 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.903882 30905 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:19.904158 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0867c6a22c2c4efd9a9babff371cc8ba" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:19.904275 29729 raft_consensus.cc:3037] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.904297 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0867c6a22c2c4efd9a9babff371cc8ba" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:19.904418 29862 raft_consensus.cc:3037] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.905346 29862 raft_consensus.cc:2454] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:19.905335 29729 raft_consensus.cc:2454] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:19.905632 29536 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:19.905777 30905 raft_consensus.cc:2781] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.905864 30905 raft_consensus.cc:686] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:19.906002 30905 consensus_queue.cc:237] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.906766 31057 raft_consensus.cc:482] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:19.906872 31057 raft_consensus.cc:504] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.907238 31057 leader_election.cc:288] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.907552 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3517a9ff83644ccb5cd4c0ebbd6e929" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:19.907619 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3517a9ff83644ccb5cd4c0ebbd6e929" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:19.907655 29600 raft_consensus.cc:2454] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:19.907716 29729 raft_consensus.cc:2454] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:19.907780 29327 catalog_manager.cc:5484] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:19.907948 29797 leader_election.cc:302] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:19.908098 31057 raft_consensus.cc:2781] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:19.908161 31057 raft_consensus.cc:482] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:19.908196 31057 raft_consensus.cc:3037] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.908919 31057 raft_consensus.cc:504] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.909287 31057 leader_election.cc:288] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:19.909560 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3517a9ff83644ccb5cd4c0ebbd6e929" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:19.909663 29600 raft_consensus.cc:3037] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.909675 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3517a9ff83644ccb5cd4c0ebbd6e929" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:19.909772 29729 raft_consensus.cc:3037] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:19.910598 29600 raft_consensus.cc:2454] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.910598 29729 raft_consensus.cc:2454] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:19.910882 29797 leader_election.cc:302] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:19.911131 31057 raft_consensus.cc:2781] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:19.911258 31057 raft_consensus.cc:686] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:19.911396 31057 consensus_queue.cc:237] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:19.912853 29327 catalog_manager.cc:5484] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:20.002943 30905 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:20.003065 30905 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.003424 30905 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:20.003751 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253ea421888249d180928ba0aef28ddd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:20.003764 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253ea421888249d180928ba0aef28ddd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:20.003877 29733 raft_consensus.cc:2454] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:20.003901 29470 raft_consensus.cc:2454] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:20.004166 29536 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:20.004350 30905 raft_consensus.cc:2781] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:20.004416 30905 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:20.004453 30905 raft_consensus.cc:3037] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.005221 30905 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.005507 30905 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:20.005831 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253ea421888249d180928ba0aef28ddd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:20.005831 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253ea421888249d180928ba0aef28ddd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:20.005949 29470 raft_consensus.cc:3037] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.005966 29733 raft_consensus.cc:3037] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.006930 29470 raft_consensus.cc:2454] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:20.006930 29733 raft_consensus.cc:2454] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:20.007228 29537 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:20.007382 30905 raft_consensus.cc:2781] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:20.007473 30905 raft_consensus.cc:686] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:20.007613 30905 consensus_queue.cc:237] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.009354 29327 catalog_manager.cc:5484] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:20.140344 31073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:20.140632 31073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:20.143704 31073 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 14:00:20.223256 29862 raft_consensus.cc:1261] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.223928 30905 consensus_queue.cc:1035] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.224445 29470 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.224678 29733 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.225337 30905 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.226331 29733 raft_consensus.cc:1261] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.226716 31057 consensus_queue.cc:1035] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.230772 30684 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.231890 29732 raft_consensus.cc:1261] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.232249 30684 consensus_queue.cc:1035] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.232302 29600 raft_consensus.cc:1261] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.235813 30707 consensus_queue.cc:1035] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.716 ms
time per row: 0.0538578 ms
I20240502 14:00:20.293434 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:20.299906 29843 tablet_service.cc:1467] Processing CreateTablet for tablet f4697431776e44a1907cf4b0865414da (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.300246 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4697431776e44a1907cf4b0865414da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.300212 29713 tablet_service.cc:1467] Processing CreateTablet for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.300457 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.301993 30852 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:20.302011 30855 tablet_bootstrap.cc:492] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:20.302209 29450 tablet_service.cc:1467] Processing CreateTablet for tablet f4697431776e44a1907cf4b0865414da (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.302484 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4697431776e44a1907cf4b0865414da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.302879 30852 tablet_bootstrap.cc:654] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.302932 30855 tablet_bootstrap.cc:654] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.304090 30855 tablet_bootstrap.cc:492] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:20.304176 30855 ts_tablet_manager.cc:1367] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:20.304631 30854 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:20.304625 30855 raft_consensus.cc:348] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.304723 30855 raft_consensus.cc:374] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.304755 30855 raft_consensus.cc:729] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.304811 29449 tablet_service.cc:1467] Processing CreateTablet for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.304872 30855 consensus_queue.cc:260] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.305032 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.305172 30855 ts_tablet_manager.cc:1398] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:20.305428 30854 tablet_bootstrap.cc:654] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.306493 30854 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:20.306561 30854 ts_tablet_manager.cc:1367] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:20.306982 30854 raft_consensus.cc:348] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.307080 30854 raft_consensus.cc:374] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.307107 30854 raft_consensus.cc:729] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.307216 30854 consensus_queue.cc:260] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.307471 30854 ts_tablet_manager.cc:1398] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:20.307608 30854 tablet_bootstrap.cc:492] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:20.308178 29843 tablet_service.cc:1467] Processing CreateTablet for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.308303 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 6767168f6f2b4e15aab7e64b0eab57c3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.308393 30854 tablet_bootstrap.cc:654] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.308424 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.308631 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6767168f6f2b4e15aab7e64b0eab57c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.311373 29582 tablet_service.cc:1467] Processing CreateTablet for tablet f4697431776e44a1907cf4b0865414da (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.311671 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4697431776e44a1907cf4b0865414da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.313488 30851 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:20.313668 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 6767168f6f2b4e15aab7e64b0eab57c3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.313899 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6767168f6f2b4e15aab7e64b0eab57c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.314345 30851 tablet_bootstrap.cc:654] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.315488 30854 tablet_bootstrap.cc:492] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:20.315631 30854 ts_tablet_manager.cc:1367] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:20.316289 30854 raft_consensus.cc:348] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.316470 30854 raft_consensus.cc:374] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.316560 30854 raft_consensus.cc:729] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.316751 30854 consensus_queue.cc:260] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.316874 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 6767168f6f2b4e15aab7e64b0eab57c3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:20.317107 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6767168f6f2b4e15aab7e64b0eab57c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.317660 30854 ts_tablet_manager.cc:1398] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:20.319190 30854 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:20.320248 30852 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:20.320346 30852 ts_tablet_manager.cc:1367] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240502 14:00:20.320770 30852 raft_consensus.cc:348] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.320868 30852 raft_consensus.cc:374] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.320905 30852 raft_consensus.cc:729] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.321015 30852 consensus_queue.cc:260] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.321254 30852 ts_tablet_manager.cc:1398] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:20.321415 30852 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:20.322244 30852 tablet_bootstrap.cc:654] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.324513 30851 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:20.324604 30851 ts_tablet_manager.cc:1367] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:20.325031 30851 raft_consensus.cc:348] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.325137 30851 raft_consensus.cc:374] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.325170 30851 raft_consensus.cc:729] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.325275 30851 consensus_queue.cc:260] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.325528 30851 ts_tablet_manager.cc:1398] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:20.326897 30852 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:20.326975 30852 ts_tablet_manager.cc:1367] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:20.327358 30852 raft_consensus.cc:348] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.327454 30852 raft_consensus.cc:374] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.327488 30852 raft_consensus.cc:729] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.327916 30851 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:20.328787 30851 tablet_bootstrap.cc:654] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.329782 30851 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:20.329881 30851 ts_tablet_manager.cc:1367] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:20.330236 30852 consensus_queue.cc:260] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.330258 30851 raft_consensus.cc:348] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.330610 30852 ts_tablet_manager.cc:1398] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:20.330356 30851 raft_consensus.cc:374] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.330922 30851 raft_consensus.cc:729] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.331080 30851 consensus_queue.cc:260] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.331369 30851 ts_tablet_manager.cc:1398] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:20.331583 30786 raft_consensus.cc:482] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:20.331676 30786 raft_consensus.cc:504] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.332043 30786 leader_election.cc:288] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:20.332757 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f251ff0bea5c4f1f8f9f1e14ca8de60d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:20.332885 29733 raft_consensus.cc:2454] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:20.333024 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f251ff0bea5c4f1f8f9f1e14ca8de60d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:20.333230 29405 leader_election.cc:302] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:20.333462 30852 tablet_bootstrap.cc:492] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
W20240502 14:00:20.333547 29405 leader_election.cc:341] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:20.333999 30786 raft_consensus.cc:2781] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:20.334051 30786 raft_consensus.cc:482] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:20.334103 30786 raft_consensus.cc:3037] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.334932 30786 raft_consensus.cc:504] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.335276 30786 leader_election.cc:288] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:20.335712 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f251ff0bea5c4f1f8f9f1e14ca8de60d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:20.335867 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f251ff0bea5c4f1f8f9f1e14ca8de60d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:20.335969 29733 raft_consensus.cc:3037] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.336236 30852 tablet_bootstrap.cc:654] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
W20240502 14:00:20.336275 29405 leader_election.cc:341] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:20.336930 29733 raft_consensus.cc:2454] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:20.337026 30854 tablet_bootstrap.cc:654] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:20.337293 29405 leader_election.cc:302] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:20.337718 30786 raft_consensus.cc:2781] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:20.337816 30786 raft_consensus.cc:686] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:20.337874 30852 tablet_bootstrap.cc:492] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:20.337953 30852 ts_tablet_manager.cc:1367] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:20.337975 30786 consensus_queue.cc:237] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.338337 30854 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:20.338436 30854 ts_tablet_manager.cc:1367] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240502 14:00:20.338385 30852 raft_consensus.cc:348] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.338479 30852 raft_consensus.cc:374] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.338531 30852 raft_consensus.cc:729] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.338688 30852 consensus_queue.cc:260] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.338882 30854 raft_consensus.cc:348] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.338972 30854 raft_consensus.cc:374] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:20.339002 30854 raft_consensus.cc:729] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:20.339128 30854 consensus_queue.cc:260] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.339668 30852 ts_tablet_manager.cc:1398] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:20.339483 30854 ts_tablet_manager.cc:1398] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:20.339834 29326 catalog_manager.cc:5484] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:20.454550 30830 raft_consensus.cc:482] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:20.454682 30830 raft_consensus.cc:504] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:20.455104 30830 leader_election.cc:288] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:20.455535 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f251ff0bea5c4f1f8f9f1e14ca8de60d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:20.455535 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f251ff0bea5c4f1f8f9f1e14ca8de60d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:20.455724 29733 raft_consensus.cc:2379] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in current term 1: Already voted for candidate 1d00d3a47c52478894cf8c66959ef7bd in this term.
I20240502 14:00:20.456001 29799 leader_election.cc:302] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:20.456178 30830 raft_consensus.cc:3037] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.457013 30830 raft_consensus.cc:2726] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:00:20.529304 30830 raft_consensus.cc:482] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:20.529433 30830 raft_consensus.cc:504] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.529785 30830 leader_election.cc:288] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:20.530184 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4697431776e44a1907cf4b0865414da" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:20.530198 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4697431776e44a1907cf4b0865414da" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:20.530316 29470 raft_consensus.cc:2454] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:20.530332 29600 raft_consensus.cc:2454] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:20.530613 29797 leader_election.cc:302] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:20.530771 30830 raft_consensus.cc:2781] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:20.530834 30830 raft_consensus.cc:482] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:20.530912 30830 raft_consensus.cc:3037] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.531689 30830 raft_consensus.cc:504] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.532025 30830 leader_election.cc:288] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:20.532346 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4697431776e44a1907cf4b0865414da" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:20.532351 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4697431776e44a1907cf4b0865414da" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:20.532456 29470 raft_consensus.cc:3037] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.532464 29600 raft_consensus.cc:3037] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.533216 29470 raft_consensus.cc:2454] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:20.533233 29600 raft_consensus.cc:2454] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:20.533515 29797 leader_election.cc:302] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:20.533650 30830 raft_consensus.cc:2781] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:20.533730 30830 raft_consensus.cc:686] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:20.533890 30830 consensus_queue.cc:237] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.535686 29326 catalog_manager.cc:5484] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:20.727102 30786 raft_consensus.cc:482] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:20.727253 30786 raft_consensus.cc:504] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.727617 30786 leader_election.cc:288] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:20.727988 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6767168f6f2b4e15aab7e64b0eab57c3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:20.728026 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6767168f6f2b4e15aab7e64b0eab57c3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:20.728117 29862 raft_consensus.cc:2454] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:20.728163 29600 raft_consensus.cc:2454] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:20.728436 29405 leader_election.cc:302] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:20.728608 30786 raft_consensus.cc:2781] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:20.728662 30786 raft_consensus.cc:482] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:20.728699 30786 raft_consensus.cc:3037] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.729511 30786 raft_consensus.cc:504] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.729835 30786 leader_election.cc:288] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:20.730134 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6767168f6f2b4e15aab7e64b0eab57c3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:20.730168 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6767168f6f2b4e15aab7e64b0eab57c3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:20.730235 29862 raft_consensus.cc:3037] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.730268 29600 raft_consensus.cc:3037] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:20.731027 29862 raft_consensus.cc:2454] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:20.731024 29600 raft_consensus.cc:2454] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:20.731284 29405 leader_election.cc:302] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:20.731472 30786 raft_consensus.cc:2781] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:20.731551 30786 raft_consensus.cc:686] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:20.731694 30786 consensus_queue.cc:237] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:20.733369 29326 catalog_manager.cc:5484] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:20.825479 30786 consensus_queue.cc:1035] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.830693 30786 consensus_queue.cc:1035] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:20.885490 31124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:20.885738 31124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:20.888998 31124 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 14:00:20.921785 30830 consensus_queue.cc:1035] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.924803 31057 consensus_queue.cc:1035] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.990725 29601 raft_consensus.cc:1261] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.991096 29864 raft_consensus.cc:1261] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:20.991947 30786 consensus_queue.cc:1035] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:20.993649 30786 consensus_queue.cc:1035] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.808 ms
time per row: 0.0564041 ms
I20240502 14:00:21.051151 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:21.056501 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 92cc4394003840a5b1d59b0fe72c5164 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.056838 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cc4394003840a5b1d59b0fe72c5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.057214 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 1089cd67cbd24551836ccb702ff44d4a (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.057277 29711 tablet_service.cc:1467] Processing CreateTablet for tablet bd91ed1ae1354a84a0ff668efdbc03d9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.057504 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1089cd67cbd24551836ccb702ff44d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.057548 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd91ed1ae1354a84a0ff668efdbc03d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.057798 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 1089cd67cbd24551836ccb702ff44d4a (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.058017 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1089cd67cbd24551836ccb702ff44d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.059198 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 92cc4394003840a5b1d59b0fe72c5164 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.059303 31187 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:21.059460 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cc4394003840a5b1d59b0fe72c5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.060132 31188 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:21.060334 31187 tablet_bootstrap.cc:654] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.060420 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 92cc4394003840a5b1d59b0fe72c5164 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.060652 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cc4394003840a5b1d59b0fe72c5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.061051 31188 tablet_bootstrap.cc:654] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.062126 29582 tablet_service.cc:1467] Processing CreateTablet for tablet bd91ed1ae1354a84a0ff668efdbc03d9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.062366 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd91ed1ae1354a84a0ff668efdbc03d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.062659 31190 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:21.063016 31191 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:21.063506 31190 tablet_bootstrap.cc:654] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.064472 31190 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:21.064527 31188 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:21.064558 31190 ts_tablet_manager.cc:1367] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:21.064602 31188 ts_tablet_manager.cc:1367] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:21.064635 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 1089cd67cbd24551836ccb702ff44d4a (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.064868 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1089cd67cbd24551836ccb702ff44d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.064989 31190 raft_consensus.cc:348] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.065017 31188 raft_consensus.cc:348] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.065099 31190 raft_consensus.cc:374] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.065155 31190 raft_consensus.cc:729] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.065270 31190 consensus_queue.cc:260] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.065623 31190 ts_tablet_manager.cc:1398] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:21.065786 31190 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:21.066174 31188 raft_consensus.cc:374] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.066273 31188 raft_consensus.cc:729] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.066431 31188 consensus_queue.cc:260] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.066609 31190 tablet_bootstrap.cc:654] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.066824 31188 ts_tablet_manager.cc:1398] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:21.067051 31188 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:21.067402 31187 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:21.067507 31187 ts_tablet_manager.cc:1367] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:21.067945 31188 tablet_bootstrap.cc:654] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.068420 31191 tablet_bootstrap.cc:654] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.069473 31187 raft_consensus.cc:348] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.069726 31187 raft_consensus.cc:374] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.069780 31187 raft_consensus.cc:729] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.069931 29843 tablet_service.cc:1467] Processing CreateTablet for tablet bd91ed1ae1354a84a0ff668efdbc03d9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.069967 31187 consensus_queue.cc:260] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.070230 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd91ed1ae1354a84a0ff668efdbc03d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.070276 31187 ts_tablet_manager.cc:1398] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:21.070513 31188 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:21.070592 31188 ts_tablet_manager.cc:1367] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:21.070982 31188 raft_consensus.cc:348] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.071075 31190 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:21.071090 31188 raft_consensus.cc:374] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.071122 31188 raft_consensus.cc:729] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.071138 31190 ts_tablet_manager.cc:1367] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:21.071517 31190 raft_consensus.cc:348] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.071617 31190 raft_consensus.cc:374] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.071648 31190 raft_consensus.cc:729] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.071768 31190 consensus_queue.cc:260] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.072021 31190 ts_tablet_manager.cc:1398] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:21.072218 31187 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:21.072852 31188 consensus_queue.cc:260] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.073130 31188 ts_tablet_manager.cc:1398] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:21.073462 31188 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:21.073539 31187 tablet_bootstrap.cc:654] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.074306 31188 tablet_bootstrap.cc:654] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.076712 31188 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:21.076807 31188 ts_tablet_manager.cc:1367] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:21.077081 31187 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:21.077169 31187 ts_tablet_manager.cc:1367] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:21.077236 31188 raft_consensus.cc:348] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.077298 31191 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:21.077347 31188 raft_consensus.cc:374] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.077376 31188 raft_consensus.cc:729] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.077397 31191 ts_tablet_manager.cc:1367] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240502 14:00:21.077497 31188 consensus_queue.cc:260] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.077626 31187 raft_consensus.cc:348] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.077742 31187 raft_consensus.cc:374] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.077776 31187 raft_consensus.cc:729] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.077850 31188 ts_tablet_manager.cc:1398] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:00:21.077885 31187 consensus_queue.cc:260] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.077975 31191 raft_consensus.cc:348] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.078173 31191 raft_consensus.cc:374] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.078205 31187 ts_tablet_manager.cc:1398] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:21.078222 31191 raft_consensus.cc:729] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.078374 31191 consensus_queue.cc:260] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.079762 31191 ts_tablet_manager.cc:1398] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:21.079932 31191 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:21.080847 31191 tablet_bootstrap.cc:654] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.081950 31191 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:21.082049 31191 ts_tablet_manager.cc:1367] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:21.082526 31191 raft_consensus.cc:348] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.082605 31191 raft_consensus.cc:374] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.082626 31191 raft_consensus.cc:729] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.082729 31191 consensus_queue.cc:260] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.082985 31191 ts_tablet_manager.cc:1398] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:21.125540 31130 raft_consensus.cc:482] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:21.125672 31130 raft_consensus.cc:504] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.126049 31130 leader_election.cc:288] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:21.126427 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cc4394003840a5b1d59b0fe72c5164" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:21.126475 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cc4394003840a5b1d59b0fe72c5164" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:21.126571 29601 raft_consensus.cc:2454] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:21.126605 29470 raft_consensus.cc:2454] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:21.126906 29668 leader_election.cc:302] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.127075 31130 raft_consensus.cc:2781] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:21.127135 31130 raft_consensus.cc:482] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:21.127179 31130 raft_consensus.cc:3037] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.128289 31130 raft_consensus.cc:504] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.128645 31130 leader_election.cc:288] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:21.128924 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cc4394003840a5b1d59b0fe72c5164" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:21.128926 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cc4394003840a5b1d59b0fe72c5164" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:21.129032 29601 raft_consensus.cc:3037] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.129024 29470 raft_consensus.cc:3037] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.129807 29601 raft_consensus.cc:2454] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:21.129948 29470 raft_consensus.cc:2454] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:21.130102 29666 leader_election.cc:302] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.130282 31130 raft_consensus.cc:2781] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:21.130367 31130 raft_consensus.cc:686] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:21.130523 31130 consensus_queue.cc:237] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.132354 29328 catalog_manager.cc:5484] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:21.140483 31130 raft_consensus.cc:482] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:21.140587 31130 raft_consensus.cc:504] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.140933 31130 leader_election.cc:288] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.141291 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1089cd67cbd24551836ccb702ff44d4a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:21.141333 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1089cd67cbd24551836ccb702ff44d4a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:21.141430 29470 raft_consensus.cc:2454] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:21.141465 29864 raft_consensus.cc:2454] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:21.141709 29668 leader_election.cc:302] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.141861 31130 raft_consensus.cc:2781] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:21.141920 31130 raft_consensus.cc:482] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:21.141965 31130 raft_consensus.cc:3037] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.142771 31130 raft_consensus.cc:504] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.143115 31130 leader_election.cc:288] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.143496 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1089cd67cbd24551836ccb702ff44d4a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:21.143493 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1089cd67cbd24551836ccb702ff44d4a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:21.143601 29470 raft_consensus.cc:3037] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.143613 29864 raft_consensus.cc:3037] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.144351 29864 raft_consensus.cc:2454] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:21.144490 29470 raft_consensus.cc:2454] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:21.144625 29667 leader_election.cc:302] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.144882 31130 raft_consensus.cc:2781] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:21.145018 31130 raft_consensus.cc:686] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:21.145151 31130 consensus_queue.cc:237] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.146957 29328 catalog_manager.cc:5484] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:21.190279 31131 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:21.190387 31131 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.190745 31131 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.191164 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91ed1ae1354a84a0ff668efdbc03d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:21.191174 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91ed1ae1354a84a0ff668efdbc03d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:21.191285 29864 raft_consensus.cc:2454] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:21.191308 29732 raft_consensus.cc:2454] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:21.191565 29536 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:21.191737 31131 raft_consensus.cc:2781] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:21.191794 31131 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:21.191845 31131 raft_consensus.cc:3037] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.192759 31131 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.193058 31131 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.193449 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91ed1ae1354a84a0ff668efdbc03d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:21.193552 29864 raft_consensus.cc:3037] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.194438 29864 raft_consensus.cc:2454] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:21.194661 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91ed1ae1354a84a0ff668efdbc03d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:21.194682 29536 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:21.194773 29732 raft_consensus.cc:3037] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.194830 31131 raft_consensus.cc:2781] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:21.195061 31131 raft_consensus.cc:686] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:21.195209 31131 consensus_queue.cc:237] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:21.195785 29732 raft_consensus.cc:2454] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:21.197216 29328 catalog_manager.cc:5484] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:00:21.347909 31204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:21.348277 31204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:21.351958 31204 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 14:00:21.438894 29600 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:21.439548 29470 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:21.440155 30910 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:21.440387 31130 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:21.441499 29864 raft_consensus.cc:1261] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:21.441802 29469 raft_consensus.cc:1261] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:21.442904 30910 consensus_queue.cc:1035] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:21.443223 30910 consensus_queue.cc:1035] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:21.443810 29862 raft_consensus.cc:1261] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:21.446720 29732 raft_consensus.cc:1261] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:21.448073 31101 consensus_queue.cc:1035] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:21.452229 31131 consensus_queue.cc:1035] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:21.459646 31129 mvcc.cc:205] Tried to move back new op lower bound from 7023240894209032192 to 7023240892951846912. Current Snapshot: MvccSnapshot[applied={T|T < 7023240894209032192}]
INSERT report
rows total: 2000
time total: 125.65 ms
time per row: 0.0628252 ms
I20240502 14:00:21.523573 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:21.528836 29450 tablet_service.cc:1467] Processing CreateTablet for tablet c35dda6845eb484886eed36e40b9231c (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.529168 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35dda6845eb484886eed36e40b9231c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.529337 29449 tablet_service.cc:1467] Processing CreateTablet for tablet bcd9a7f130364a46b4a48f96bb28f7a6 (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.529606 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd9a7f130364a46b4a48f96bb28f7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.529742 29843 tablet_service.cc:1467] Processing CreateTablet for tablet c35dda6845eb484886eed36e40b9231c (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.530011 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35dda6845eb484886eed36e40b9231c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.531181 29582 tablet_service.cc:1467] Processing CreateTablet for tablet c35dda6845eb484886eed36e40b9231c (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.531468 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35dda6845eb484886eed36e40b9231c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.532136 31191 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:21.532114 29449 tablet_service.cc:1467] Processing CreateTablet for tablet add1fc32951945488dfb045e56dcecc9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.532352 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add1fc32951945488dfb045e56dcecc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.533011 31191 tablet_bootstrap.cc:654] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.533255 31190 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:21.533321 31187 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:21.534022 31187 tablet_bootstrap.cc:654] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.534022 31190 tablet_bootstrap.cc:654] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.531181 29581 tablet_service.cc:1467] Processing CreateTablet for tablet bcd9a7f130364a46b4a48f96bb28f7a6 (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.534499 29843 tablet_service.cc:1467] Processing CreateTablet for tablet add1fc32951945488dfb045e56dcecc9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.534705 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add1fc32951945488dfb045e56dcecc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.534711 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd9a7f130364a46b4a48f96bb28f7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.535337 29713 tablet_service.cc:1467] Processing CreateTablet for tablet bcd9a7f130364a46b4a48f96bb28f7a6 (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.535640 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd9a7f130364a46b4a48f96bb28f7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.536680 31187 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:21.536759 31187 ts_tablet_manager.cc:1367] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:21.537052 31187 raft_consensus.cc:348] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.537145 31187 raft_consensus.cc:374] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.537176 31187 raft_consensus.cc:729] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.537273 31187 consensus_queue.cc:260] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.537531 31187 ts_tablet_manager.cc:1398] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:21.537580 31188 tablet_bootstrap.cc:492] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:21.537688 31187 tablet_bootstrap.cc:492] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:21.537765 29712 tablet_service.cc:1467] Processing CreateTablet for tablet add1fc32951945488dfb045e56dcecc9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:21.537959 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add1fc32951945488dfb045e56dcecc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:21.538429 31188 tablet_bootstrap.cc:654] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.539392 31187 tablet_bootstrap.cc:654] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.540367 31188 tablet_bootstrap.cc:492] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:21.540450 31188 ts_tablet_manager.cc:1367] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:21.540791 31191 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:21.540848 31191 ts_tablet_manager.cc:1367] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:00:21.540809 31188 raft_consensus.cc:348] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.540892 31188 raft_consensus.cc:374] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.540913 31188 raft_consensus.cc:729] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.540999 31188 consensus_queue.cc:260] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.541183 31191 raft_consensus.cc:348] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.541241 31188 ts_tablet_manager.cc:1398] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:21.541261 31191 raft_consensus.cc:374] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.541282 31191 raft_consensus.cc:729] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.541388 31191 consensus_queue.cc:260] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.541518 31188 tablet_bootstrap.cc:492] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:21.541611 31187 tablet_bootstrap.cc:492] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:21.541651 31191 ts_tablet_manager.cc:1398] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:21.541677 31187 ts_tablet_manager.cc:1367] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:21.542078 31187 raft_consensus.cc:348] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.542173 31187 raft_consensus.cc:374] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.542203 31187 raft_consensus.cc:729] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.542309 31187 consensus_queue.cc:260] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.542479 31188 tablet_bootstrap.cc:654] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.542569 31187 ts_tablet_manager.cc:1398] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:21.542708 31187 tablet_bootstrap.cc:492] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:21.543588 31187 tablet_bootstrap.cc:654] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.544492 31191 tablet_bootstrap.cc:492] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:21.544860 31187 tablet_bootstrap.cc:492] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:21.544950 31187 ts_tablet_manager.cc:1367] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:21.545269 31188 tablet_bootstrap.cc:492] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:21.545341 31188 ts_tablet_manager.cc:1367] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:00:21.545365 31191 tablet_bootstrap.cc:654] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.545713 31187 raft_consensus.cc:348] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.545809 31187 raft_consensus.cc:374] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.545845 31187 raft_consensus.cc:729] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.545876 31188 raft_consensus.cc:348] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.545990 31188 raft_consensus.cc:374] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.545958 31187 consensus_queue.cc:260] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.546036 31188 raft_consensus.cc:729] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.546171 31188 consensus_queue.cc:260] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.546272 31187 ts_tablet_manager.cc:1398] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:21.546453 31188 ts_tablet_manager.cc:1398] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:21.547201 31191 tablet_bootstrap.cc:492] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:21.547282 31191 ts_tablet_manager.cc:1367] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:21.547282 31190 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:21.547382 31190 ts_tablet_manager.cc:1367] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240502 14:00:21.547659 31191 raft_consensus.cc:348] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.547673 31190 raft_consensus.cc:348] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.547745 31190 raft_consensus.cc:374] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.547742 31191 raft_consensus.cc:374] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.547765 31190 raft_consensus.cc:729] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.547780 31191 raft_consensus.cc:729] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.547863 31190 consensus_queue.cc:260] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.547883 31191 consensus_queue.cc:260] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.548115 31190 ts_tablet_manager.cc:1398] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:21.548147 31191 ts_tablet_manager.cc:1398] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:21.548933 31190 tablet_bootstrap.cc:492] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:21.549656 31190 tablet_bootstrap.cc:654] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:21.550902 31190 tablet_bootstrap.cc:492] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:21.550987 31190 ts_tablet_manager.cc:1367] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:21.551364 31190 raft_consensus.cc:348] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.551471 31190 raft_consensus.cc:374] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:21.551503 31190 raft_consensus.cc:729] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:21.551596 31190 consensus_queue.cc:260] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.553408 31190 ts_tablet_manager.cc:1398] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:21.664240 31239 raft_consensus.cc:482] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:21.664376 31239 raft_consensus.cc:504] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.664809 31239 leader_election.cc:288] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.665174 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add1fc32951945488dfb045e56dcecc9" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:21.665237 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add1fc32951945488dfb045e56dcecc9" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:21.665325 29469 raft_consensus.cc:2454] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:21.665354 29864 raft_consensus.cc:2454] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:21.665613 29668 leader_election.cc:302] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.665786 31239 raft_consensus.cc:2781] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:21.665858 31239 raft_consensus.cc:482] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:21.665887 31239 raft_consensus.cc:3037] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.666767 31239 raft_consensus.cc:504] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.667158 31239 leader_election.cc:288] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.667464 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add1fc32951945488dfb045e56dcecc9" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:21.667464 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add1fc32951945488dfb045e56dcecc9" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:21.667572 29469 raft_consensus.cc:3037] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.667582 29864 raft_consensus.cc:3037] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.668299 29864 raft_consensus.cc:2454] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:21.668305 29469 raft_consensus.cc:2454] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:21.668579 29668 leader_election.cc:302] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.668771 31239 raft_consensus.cc:2781] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:21.668864 31239 raft_consensus.cc:686] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:21.668999 31239 consensus_queue.cc:237] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:21.670823 29327 catalog_manager.cc:5484] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:21.691104 31101 raft_consensus.cc:482] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:21.691216 31101 raft_consensus.cc:504] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.691617 31101 leader_election.cc:288] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.691958 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35dda6845eb484886eed36e40b9231c" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:21.691943 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35dda6845eb484886eed36e40b9231c" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:21.692090 29469 raft_consensus.cc:2454] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:21.692090 29864 raft_consensus.cc:2454] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:21.692368 29537 leader_election.cc:302] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:21.692515 31101 raft_consensus.cc:2781] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:21.692571 31101 raft_consensus.cc:482] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:21.692612 31101 raft_consensus.cc:3037] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.693544 31101 raft_consensus.cc:504] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.693890 31101 leader_election.cc:288] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:21.694192 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35dda6845eb484886eed36e40b9231c" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:21.694285 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35dda6845eb484886eed36e40b9231c" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:21.694335 29469 raft_consensus.cc:3037] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.694409 29864 raft_consensus.cc:3037] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.695344 29469 raft_consensus.cc:2454] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:21.695344 29864 raft_consensus.cc:2454] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:21.695629 29537 leader_election.cc:302] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:21.695845 31101 raft_consensus.cc:2781] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:21.695914 31101 raft_consensus.cc:686] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:21.696054 31101 consensus_queue.cc:237] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:21.697753 29327 catalog_manager.cc:5484] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:21.845347 31131 raft_consensus.cc:482] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:21.845500 31131 raft_consensus.cc:504] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.845896 31131 leader_election.cc:288] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:21.846374 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9a7f130364a46b4a48f96bb28f7a6" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:21.846462 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9a7f130364a46b4a48f96bb28f7a6" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:21.846510 29732 raft_consensus.cc:2454] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:21.846633 29469 raft_consensus.cc:2454] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:21.846880 29536 leader_election.cc:302] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.847044 31131 raft_consensus.cc:2781] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:21.847101 31131 raft_consensus.cc:482] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:21.847168 31131 raft_consensus.cc:3037] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.847951 31131 raft_consensus.cc:504] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.848285 31131 leader_election.cc:288] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:21.848589 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9a7f130364a46b4a48f96bb28f7a6" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:21.848620 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9a7f130364a46b4a48f96bb28f7a6" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:21.848687 29732 raft_consensus.cc:3037] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.848716 29469 raft_consensus.cc:3037] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:21.849448 29732 raft_consensus.cc:2454] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:21.849540 29469 raft_consensus.cc:2454] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:21.849706 29536 leader_election.cc:302] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:21.849895 31131 raft_consensus.cc:2781] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:21.850016 31131 raft_consensus.cc:686] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:21.850183 31131 consensus_queue.cc:237] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:21.851928 29327 catalog_manager.cc:5484] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:00:22.010830 31253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:22.011128 31253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:22.014529 31253 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240502 14:00:22.102308 29864 raft_consensus.cc:1261] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.102597 29862 raft_consensus.cc:1261] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.102902 31239 consensus_queue.cc:1035] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.102999 29469 raft_consensus.cc:1261] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.104051 29733 raft_consensus.cc:1261] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.104699 29470 raft_consensus.cc:1261] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.104702 31101 consensus_queue.cc:1035] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.105028 31101 consensus_queue.cc:1035] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.105578 31131 consensus_queue.cc:1035] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.106918 29469 raft_consensus.cc:1261] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.107360 31239 consensus_queue.cc:1035] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.109640 31131 consensus_queue.cc:1035] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.119863 31289 mvcc.cc:205] Tried to move back new op lower bound from 7023240896925802496 to 7023240895157407744. Current Snapshot: MvccSnapshot[applied={T|T < 7023240896925802496}]
I20240502 14:00:22.121269 31287 mvcc.cc:205] Tried to move back new op lower bound from 7023240896938385408 to 7023240895268241408. Current Snapshot: MvccSnapshot[applied={T|T < 7023240896924078080}]
I20240502 14:00:22.158938 31295 mvcc.cc:205] Tried to move back new op lower bound from 7023240896938385408 to 7023240895268241408. Current Snapshot: MvccSnapshot[applied={T|T < 7023240896924078080}]
INSERT report
rows total: 2000
time total: 119.471 ms
time per row: 0.0597356 ms
I20240502 14:00:22.181922 29320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:22.187217 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 93e77c40b5674865ba6ccce626545ec7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.187462 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 93e77c40b5674865ba6ccce626545ec7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.187574 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e77c40b5674865ba6ccce626545ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.187747 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e77c40b5674865ba6ccce626545ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.187695 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 9bdf925a9c4d456fb3b4ef448a2154c6 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.187947 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdf925a9c4d456fb3b4ef448a2154c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.187916 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 93e77c40b5674865ba6ccce626545ec7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.188148 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e77c40b5674865ba6ccce626545ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.188627 29844 tablet_service.cc:1467] Processing CreateTablet for tablet b19374a33aaf41069e98fd035d2b9f00 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.188856 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19374a33aaf41069e98fd035d2b9f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.190609 31319 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:22.190696 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 9bdf925a9c4d456fb3b4ef448a2154c6 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.190996 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdf925a9c4d456fb3b4ef448a2154c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.191135 29582 tablet_service.cc:1467] Processing CreateTablet for tablet b19374a33aaf41069e98fd035d2b9f00 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.191210 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 9bdf925a9c4d456fb3b4ef448a2154c6 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.191373 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19374a33aaf41069e98fd035d2b9f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.191534 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdf925a9c4d456fb3b4ef448a2154c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.193140 31318 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:22.193656 29449 tablet_service.cc:1467] Processing CreateTablet for tablet b19374a33aaf41069e98fd035d2b9f00 (DEFAULT_TABLE table=rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.193951 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19374a33aaf41069e98fd035d2b9f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.194253 31318 tablet_bootstrap.cc:654] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.194335 31317 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:22.195214 31317 tablet_bootstrap.cc:654] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.195837 31319 tablet_bootstrap.cc:654] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.196060 31322 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:22.196410 31317 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:22.196527 31317 ts_tablet_manager.cc:1367] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:22.197106 31322 tablet_bootstrap.cc:654] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.197017 31317 raft_consensus.cc:348] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.197166 31317 raft_consensus.cc:374] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.197520 31319 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:22.197499 31317 raft_consensus.cc:729] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.197610 31319 ts_tablet_manager.cc:1367] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:22.197678 31317 consensus_queue.cc:260] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.198046 31317 ts_tablet_manager.cc:1398] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:22.198074 31319 raft_consensus.cc:348] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.198168 31319 raft_consensus.cc:374] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.198200 31319 raft_consensus.cc:729] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.198226 31317 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:22.198323 31319 consensus_queue.cc:260] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.198422 31322 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:22.198500 31322 ts_tablet_manager.cc:1367] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:22.198596 31319 ts_tablet_manager.cc:1398] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:22.198751 31319 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:22.198990 31322 raft_consensus.cc:348] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.199086 31322 raft_consensus.cc:374] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.199115 31322 raft_consensus.cc:729] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.199193 31317 tablet_bootstrap.cc:654] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.199254 31322 consensus_queue.cc:260] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.199538 31322 ts_tablet_manager.cc:1398] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.200289 31318 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:22.200384 31318 ts_tablet_manager.cc:1367] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:22.200822 31318 raft_consensus.cc:348] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.200927 31318 raft_consensus.cc:374] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.200971 31318 raft_consensus.cc:729] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.201084 31318 consensus_queue.cc:260] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.201257 31317 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:22.201332 31317 ts_tablet_manager.cc:1367] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:22.201382 31318 ts_tablet_manager.cc:1398] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.201552 31318 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:22.201741 31317 raft_consensus.cc:348] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.201830 31317 raft_consensus.cc:374] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.201861 31317 raft_consensus.cc:729] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.201983 31317 consensus_queue.cc:260] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.202245 31317 ts_tablet_manager.cc:1398] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.202414 31317 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:22.202456 31318 tablet_bootstrap.cc:654] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.203366 31317 tablet_bootstrap.cc:654] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.204519 31319 tablet_bootstrap.cc:654] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.205904 31318 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:22.205992 31317 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:22.206028 31318 ts_tablet_manager.cc:1367] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:22.206087 31317 ts_tablet_manager.cc:1367] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:22.206102 31319 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:22.206192 31319 ts_tablet_manager.cc:1367] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:22.206537 31317 raft_consensus.cc:348] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.206552 31318 raft_consensus.cc:348] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.206669 31317 raft_consensus.cc:374] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.206669 31318 raft_consensus.cc:374] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.206712 31317 raft_consensus.cc:729] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.206712 31318 raft_consensus.cc:729] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.206863 31317 consensus_queue.cc:260] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.206828 31319 raft_consensus.cc:348] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.206900 31318 consensus_queue.cc:260] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.207033 31319 raft_consensus.cc:374] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.207074 31319 raft_consensus.cc:729] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.207178 31317 ts_tablet_manager.cc:1398] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.207320 31318 ts_tablet_manager.cc:1398] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.207803 31319 consensus_queue.cc:260] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.208103 31319 ts_tablet_manager.cc:1398] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:22.208335 31319 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:22.209662 31319 tablet_bootstrap.cc:654] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.210826 31319 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:22.210932 31319 ts_tablet_manager.cc:1367] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:22.211342 31319 raft_consensus.cc:348] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.211436 31319 raft_consensus.cc:374] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.211467 31319 raft_consensus.cc:729] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.211580 31319 consensus_queue.cc:260] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.211817 31319 ts_tablet_manager.cc:1398] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:22.222702 31239 raft_consensus.cc:482] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:22.222880 31239 raft_consensus.cc:504] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.223354 31239 leader_election.cc:288] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:22.223695 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e77c40b5674865ba6ccce626545ec7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:22.223724 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e77c40b5674865ba6ccce626545ec7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:22.223845 29600 raft_consensus.cc:2454] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:22.223845 29862 raft_consensus.cc:2454] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:22.224153 29667 leader_election.cc:302] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:22.224402 31239 raft_consensus.cc:2781] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:22.224464 31239 raft_consensus.cc:482] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:22.224500 31239 raft_consensus.cc:3037] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.225484 31239 raft_consensus.cc:504] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.225806 31239 leader_election.cc:288] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:22.226089 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e77c40b5674865ba6ccce626545ec7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:22.226199 29600 raft_consensus.cc:3037] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.226166 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e77c40b5674865ba6ccce626545ec7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:22.226291 29862 raft_consensus.cc:3037] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.226989 29600 raft_consensus.cc:2454] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:22.227231 29862 raft_consensus.cc:2454] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:22.227252 29666 leader_election.cc:302] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:22.227402 31239 raft_consensus.cc:2781] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:22.227478 31239 raft_consensus.cc:686] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:22.227618 31239 consensus_queue.cc:237] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.229629 29328 catalog_manager.cc:5484] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:22.233491 31101 raft_consensus.cc:482] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:22.233598 31101 raft_consensus.cc:504] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.233953 31101 leader_election.cc:288] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:22.234311 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19374a33aaf41069e98fd035d2b9f00" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:22.234334 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19374a33aaf41069e98fd035d2b9f00" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:22.234437 29862 raft_consensus.cc:2454] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:22.234478 29470 raft_consensus.cc:2454] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:22.234743 29536 leader_election.cc:302] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:22.234941 31101 raft_consensus.cc:2781] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:22.234994 31101 raft_consensus.cc:482] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:22.235021 31101 raft_consensus.cc:3037] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.235806 31101 raft_consensus.cc:504] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.236137 31101 leader_election.cc:288] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:22.236456 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19374a33aaf41069e98fd035d2b9f00" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:22.236454 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19374a33aaf41069e98fd035d2b9f00" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:22.236552 29862 raft_consensus.cc:3037] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.236569 29470 raft_consensus.cc:3037] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.237330 29862 raft_consensus.cc:2454] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:22.237354 29470 raft_consensus.cc:2454] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:22.237634 29536 leader_election.cc:302] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:22.237785 31101 raft_consensus.cc:2781] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:22.237877 31101 raft_consensus.cc:686] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:22.238034 31101 consensus_queue.cc:237] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.239861 29328 catalog_manager.cc:5484] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:22.449620 31239 raft_consensus.cc:482] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:22.449750 31239 raft_consensus.cc:504] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.450172 31239 leader_election.cc:288] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:22.450574 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:22.450620 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:22.450714 29862 raft_consensus.cc:2454] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:22.450753 29600 raft_consensus.cc:2454] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:22.450992 29667 leader_election.cc:302] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:22.451184 31239 raft_consensus.cc:2781] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:22.451261 31239 raft_consensus.cc:482] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:22.451306 31239 raft_consensus.cc:3037] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.452104 31239 raft_consensus.cc:504] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.452672 31239 leader_election.cc:288] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:22.452733 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:22.452802 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:22.452849 29862 raft_consensus.cc:3037] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.452929 29600 raft_consensus.cc:3037] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.453858 29600 raft_consensus.cc:2454] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:22.453862 29862 raft_consensus.cc:2454] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:22.454128 29666 leader_election.cc:302] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:22.454277 31239 raft_consensus.cc:2781] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:22.454363 31239 raft_consensus.cc:686] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:22.454512 31239 consensus_queue.cc:237] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.456399 29328 catalog_manager.cc:5484] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:22.605111 31333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:22.605347 31333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:22.608752 31333 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240502 14:00:22.631879 31239 consensus_queue.cc:1035] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.635360 31239 consensus_queue.cc:1035] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.642710 31101 consensus_queue.cc:1035] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.651337 31101 consensus_queue.cc:1035] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.694619 29600 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.695267 30910 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:22.695730 29864 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:22.697031 31239 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.284 ms
time per row: 0.0516418 ms
I20240502 14:00:22.757974 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:22.763293 29449 tablet_service.cc:1467] Processing CreateTablet for tablet b1323869da8c4249b727852cb1a454b3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.763651 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1323869da8c4249b727852cb1a454b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.763598 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 7808510875f441fd9814121434f534e0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.763886 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7808510875f441fd9814121434f534e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.763825 29582 tablet_service.cc:1467] Processing CreateTablet for tablet b1323869da8c4249b727852cb1a454b3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.763901 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 7808510875f441fd9814121434f534e0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.764091 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1323869da8c4249b727852cb1a454b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.764108 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7808510875f441fd9814121434f534e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.766150 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed302438544dcf9c7e90b6a98576b3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.766379 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed302438544dcf9c7e90b6a98576b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.766726 29712 tablet_service.cc:1467] Processing CreateTablet for tablet b1323869da8c4249b727852cb1a454b3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.766819 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 7808510875f441fd9814121434f534e0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.767030 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1323869da8c4249b727852cb1a454b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.767056 31394 tablet_bootstrap.cc:492] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:22.767436 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7808510875f441fd9814121434f534e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.768038 31394 tablet_bootstrap.cc:654] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.769297 31398 tablet_bootstrap.cc:492] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:22.769315 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed302438544dcf9c7e90b6a98576b3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.769558 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed302438544dcf9c7e90b6a98576b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.769779 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed302438544dcf9c7e90b6a98576b3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:22.770028 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed302438544dcf9c7e90b6a98576b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.770345 31398 tablet_bootstrap.cc:654] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.771597 31395 tablet_bootstrap.cc:492] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:22.771611 31398 tablet_bootstrap.cc:492] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:22.771682 31398 ts_tablet_manager.cc:1367] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:22.771920 31396 tablet_bootstrap.cc:492] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:22.772145 31398 raft_consensus.cc:348] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.772264 31398 raft_consensus.cc:374] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.772296 31398 raft_consensus.cc:729] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.772415 31398 consensus_queue.cc:260] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.772657 31395 tablet_bootstrap.cc:654] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.772723 31398 ts_tablet_manager.cc:1398] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.772908 31396 tablet_bootstrap.cc:654] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.773010 31398 tablet_bootstrap.cc:492] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:22.773903 31394 tablet_bootstrap.cc:492] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:22.773969 31398 tablet_bootstrap.cc:654] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.774001 31394 ts_tablet_manager.cc:1367] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:22.774016 31396 tablet_bootstrap.cc:492] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:22.774139 31396 ts_tablet_manager.cc:1367] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:22.774466 31394 raft_consensus.cc:348] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.774581 31394 raft_consensus.cc:374] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.774627 31394 raft_consensus.cc:729] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.774660 31395 tablet_bootstrap.cc:492] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:22.774749 31395 ts_tablet_manager.cc:1367] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:22.774756 31394 consensus_queue.cc:260] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.774739 31396 raft_consensus.cc:348] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.774946 31396 raft_consensus.cc:374] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.775017 31396 raft_consensus.cc:729] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.775067 31394 ts_tablet_manager.cc:1398] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.775157 31398 tablet_bootstrap.cc:492] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:22.775174 31396 consensus_queue.cc:260] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.775233 31398 ts_tablet_manager.cc:1367] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:22.775240 31394 tablet_bootstrap.cc:492] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:22.775350 31395 raft_consensus.cc:348] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:22.775441 31395 raft_consensus.cc:374] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.775471 31395 raft_consensus.cc:729] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.775494 31396 ts_tablet_manager.cc:1398] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.775583 31395 consensus_queue.cc:260] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:22.775642 31396 tablet_bootstrap.cc:492] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:22.775743 31398 raft_consensus.cc:348] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:22.775839 31398 raft_consensus.cc:374] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.775843 31395 ts_tablet_manager.cc:1398] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:22.775869 31398 raft_consensus.cc:729] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.775980 31398 consensus_queue.cc:260] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:22.776050 31394 tablet_bootstrap.cc:654] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.776243 31398 ts_tablet_manager.cc:1398] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.776496 31396 tablet_bootstrap.cc:654] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.777329 31398 tablet_bootstrap.cc:492] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:22.778194 31398 tablet_bootstrap.cc:654] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.779587 31398 tablet_bootstrap.cc:492] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:22.777174 31394 tablet_bootstrap.cc:492] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:22.779666 31398 ts_tablet_manager.cc:1367] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:22.779685 31394 ts_tablet_manager.cc:1367] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:22.780071 31398 raft_consensus.cc:348] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.780210 31398 raft_consensus.cc:374] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.780251 31398 raft_consensus.cc:729] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.780365 31398 consensus_queue.cc:260] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.780654 31398 ts_tablet_manager.cc:1398] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:22.780844 31394 raft_consensus.cc:348] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:22.781049 31394 raft_consensus.cc:374] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.781136 31394 raft_consensus.cc:729] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.781365 31394 consensus_queue.cc:260] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:22.781658 31394 ts_tablet_manager.cc:1398] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:22.781875 31394 tablet_bootstrap.cc:492] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:22.782665 31394 tablet_bootstrap.cc:654] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.783156 31396 tablet_bootstrap.cc:492] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:22.783285 31396 ts_tablet_manager.cc:1367] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:22.783871 31394 tablet_bootstrap.cc:492] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:22.783951 31394 ts_tablet_manager.cc:1367] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:22.784348 31394 raft_consensus.cc:348] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.784430 31394 raft_consensus.cc:374] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.784464 31394 raft_consensus.cc:729] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.784456 31396 raft_consensus.cc:348] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.784559 31396 raft_consensus.cc:374] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.784605 31396 raft_consensus.cc:729] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.784592 31394 consensus_queue.cc:260] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.784722 31396 consensus_queue.cc:260] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.784837 31394 ts_tablet_manager.cc:1398] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:22.785058 31396 ts_tablet_manager.cc:1398] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:22.821913 31277 raft_consensus.cc:482] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:22.822050 31277 raft_consensus.cc:504] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.822508 31277 leader_election.cc:288] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:22.822865 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1323869da8c4249b727852cb1a454b3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:22.822906 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1323869da8c4249b727852cb1a454b3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:22.823024 29601 raft_consensus.cc:2454] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:22.823024 29732 raft_consensus.cc:2454] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:22.823348 29404 leader_election.cc:302] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:22.823531 31277 raft_consensus.cc:2781] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:22.823594 31277 raft_consensus.cc:482] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:22.823643 31277 raft_consensus.cc:3037] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.824491 31277 raft_consensus.cc:504] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.824802 31277 leader_election.cc:288] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:22.825114 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1323869da8c4249b727852cb1a454b3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:22.825132 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1323869da8c4249b727852cb1a454b3" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:22.825219 29601 raft_consensus.cc:3037] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.825241 29732 raft_consensus.cc:3037] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.825956 29601 raft_consensus.cc:2454] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:22.825995 29732 raft_consensus.cc:2454] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:22.826274 29405 leader_election.cc:302] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:22.826417 31277 raft_consensus.cc:2781] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:22.826654 31277 raft_consensus.cc:686] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:22.826802 31277 consensus_queue.cc:237] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:22.828688 29320 catalog_manager.cc:5484] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:22.866541 31391 raft_consensus.cc:482] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:22.866690 31391 raft_consensus.cc:504] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.867223 31391 leader_election.cc:288] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:22.867578 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed302438544dcf9c7e90b6a98576b3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:22.867604 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed302438544dcf9c7e90b6a98576b3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:22.867709 29470 raft_consensus.cc:2454] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:22.867734 29732 raft_consensus.cc:2454] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:22.867997 29537 leader_election.cc:302] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:22.868170 31391 raft_consensus.cc:2781] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:22.868227 31391 raft_consensus.cc:482] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:22.868278 31391 raft_consensus.cc:3037] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.869141 31391 raft_consensus.cc:504] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.869454 31391 leader_election.cc:288] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:22.869733 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed302438544dcf9c7e90b6a98576b3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:22.869833 29470 raft_consensus.cc:3037] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.869813 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed302438544dcf9c7e90b6a98576b3" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:22.869931 29732 raft_consensus.cc:3037] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.870590 29470 raft_consensus.cc:2454] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:22.870654 29732 raft_consensus.cc:2454] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:22.870890 29537 leader_election.cc:302] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:22.871043 31391 raft_consensus.cc:2781] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:22.871129 31391 raft_consensus.cc:686] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:22.871270 31391 consensus_queue.cc:237] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:22.873137 29320 catalog_manager.cc:5484] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:23.070912 31277 raft_consensus.cc:482] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:23.071049 31277 raft_consensus.cc:504] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.071477 31277 leader_election.cc:288] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:23.071745 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808510875f441fd9814121434f534e0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:23.071750 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808510875f441fd9814121434f534e0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:23.071862 29733 raft_consensus.cc:2454] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.071882 29862 raft_consensus.cc:2454] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.072132 29405 leader_election.cc:302] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:23.072300 31277 raft_consensus.cc:2781] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:23.072357 31277 raft_consensus.cc:482] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:23.072409 31277 raft_consensus.cc:3037] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.073272 31277 raft_consensus.cc:504] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.073572 31277 leader_election.cc:288] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:23.073838 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808510875f441fd9814121434f534e0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:23.073948 29733 raft_consensus.cc:3037] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.073900 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808510875f441fd9814121434f534e0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:23.074025 29862 raft_consensus.cc:3037] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.074976 29733 raft_consensus.cc:2454] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.074976 29862 raft_consensus.cc:2454] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.075245 29405 leader_election.cc:302] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:23.075521 31277 raft_consensus.cc:2781] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:23.075681 31277 raft_consensus.cc:686] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:23.075819 31277 consensus_queue.cc:237] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.077831 29320 catalog_manager.cc:5484] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:00:23.242936 31412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:23.243181 31412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:23.246225 31412 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240502 14:00:23.275094 31391 consensus_queue.cc:1035] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.278939 31131 consensus_queue.cc:1035] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.329761 31407 consensus_queue.cc:1035] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.335389 31277 consensus_queue.cc:1035] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.341521 29862 raft_consensus.cc:1261] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.341923 31277 consensus_queue.cc:1035] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.351785 29733 raft_consensus.cc:1261] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.352250 31407 consensus_queue.cc:1035] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.759 ms
time per row: 0.0568793 ms
I20240502 14:00:23.408031 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:23.413314 29582 tablet_service.cc:1467] Processing CreateTablet for tablet a7cc49f0227d491ba7a8df281856b8f8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.413479 29842 tablet_service.cc:1467] Processing CreateTablet for tablet a7cc49f0227d491ba7a8df281856b8f8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.413636 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cc49f0227d491ba7a8df281856b8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.413777 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cc49f0227d491ba7a8df281856b8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.414125 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 7650009dd8954f11ab2ecafef765692c (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.414347 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7650009dd8954f11ab2ecafef765692c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.416079 29711 tablet_service.cc:1467] Processing CreateTablet for tablet c687dbe084f7469fa0167fc769b5de65 (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.416154 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 7650009dd8954f11ab2ecafef765692c (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.416322 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c687dbe084f7469fa0167fc769b5de65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.416484 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7650009dd8954f11ab2ecafef765692c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.416785 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 7650009dd8954f11ab2ecafef765692c (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.416997 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7650009dd8954f11ab2ecafef765692c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.417112 29451 tablet_service.cc:1467] Processing CreateTablet for tablet a7cc49f0227d491ba7a8df281856b8f8 (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.417367 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cc49f0227d491ba7a8df281856b8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.418617 31465 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:23.418700 29842 tablet_service.cc:1467] Processing CreateTablet for tablet c687dbe084f7469fa0167fc769b5de65 (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.418915 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c687dbe084f7469fa0167fc769b5de65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.419090 29450 tablet_service.cc:1467] Processing CreateTablet for tablet c687dbe084f7469fa0167fc769b5de65 (DEFAULT_TABLE table=rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.419307 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c687dbe084f7469fa0167fc769b5de65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.419538 31465 tablet_bootstrap.cc:654] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.420403 31468 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:23.420527 31464 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:23.421185 31466 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:23.421532 31465 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:23.421608 31465 ts_tablet_manager.cc:1367] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:23.422122 31465 raft_consensus.cc:348] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.422257 31465 raft_consensus.cc:374] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.422281 31466 tablet_bootstrap.cc:654] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.422294 31465 raft_consensus.cc:729] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.422425 31465 consensus_queue.cc:260] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.423206 31468 tablet_bootstrap.cc:654] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.423450 31465 ts_tablet_manager.cc:1398] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:23.423630 31464 tablet_bootstrap.cc:654] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.423698 31465 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:23.424762 31466 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:23.424855 31466 ts_tablet_manager.cc:1367] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:23.424654 31465 tablet_bootstrap.cc:654] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.425364 31464 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:23.425499 31464 ts_tablet_manager.cc:1367] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:23.425519 31468 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:23.425602 31468 ts_tablet_manager.cc:1367] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:23.426066 31468 raft_consensus.cc:348] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.426195 31468 raft_consensus.cc:374] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.426236 31468 raft_consensus.cc:729] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.426429 31465 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:23.426504 31465 ts_tablet_manager.cc:1367] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:23.426726 31468 consensus_queue.cc:260] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.426090 31464 raft_consensus.cc:348] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.426923 31465 raft_consensus.cc:348] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.427012 31465 raft_consensus.cc:374] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.427040 31465 raft_consensus.cc:729] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.427157 31468 ts_tablet_manager.cc:1398] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:00:23.427155 31465 consensus_queue.cc:260] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.427417 31465 ts_tablet_manager.cc:1398] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:23.427557 31468 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:23.427729 31464 raft_consensus.cc:374] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.427778 31464 raft_consensus.cc:729] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.427971 31466 raft_consensus.cc:348] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.428068 31466 raft_consensus.cc:374] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.428102 31466 raft_consensus.cc:729] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.428251 31464 consensus_queue.cc:260] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.429210 31468 tablet_bootstrap.cc:654] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.429370 31466 consensus_queue.cc:260] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.429695 31466 ts_tablet_manager.cc:1398] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:23.429838 31464 ts_tablet_manager.cc:1398] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:23.430066 31464 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:23.430861 31466 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:23.432353 31464 tablet_bootstrap.cc:654] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.433164 31466 tablet_bootstrap.cc:654] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.434072 31464 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:23.434087 31468 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:23.434152 31468 ts_tablet_manager.cc:1367] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:23.434350 31464 ts_tablet_manager.cc:1367] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:23.434546 31468 raft_consensus.cc:348] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.434653 31468 raft_consensus.cc:374] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.434700 31468 raft_consensus.cc:729] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.434826 31468 consensus_queue.cc:260] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.434816 31464 raft_consensus.cc:348] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.434947 31464 raft_consensus.cc:374] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.434990 31464 raft_consensus.cc:729] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.435106 31464 consensus_queue.cc:260] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.435132 31468 ts_tablet_manager.cc:1398] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:23.436056 31466 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:23.436187 31466 ts_tablet_manager.cc:1367] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:23.436709 31466 raft_consensus.cc:348] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.437050 31464 ts_tablet_manager.cc:1398] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:23.437177 31466 raft_consensus.cc:374] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.437255 31466 raft_consensus.cc:729] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.437386 31466 consensus_queue.cc:260] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.437494 31461 raft_consensus.cc:482] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:23.437572 31461 raft_consensus.cc:504] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.437808 31464 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:23.437896 31461 leader_election.cc:288] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:23.438472 31466 ts_tablet_manager.cc:1398] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:23.438464 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cc49f0227d491ba7a8df281856b8f8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:23.438577 29862 raft_consensus.cc:2454] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.438895 29405 leader_election.cc:302] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:23.439214 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cc49f0227d491ba7a8df281856b8f8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:23.439289 31464 tablet_bootstrap.cc:654] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.439340 29601 raft_consensus.cc:2454] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.439903 31461 raft_consensus.cc:2781] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:23.439962 31461 raft_consensus.cc:482] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:23.440001 31461 raft_consensus.cc:3037] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.440450 31464 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:23.440522 31464 ts_tablet_manager.cc:1367] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:23.440985 31464 raft_consensus.cc:348] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.441098 31464 raft_consensus.cc:374] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.441061 31461 raft_consensus.cc:504] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.441129 31464 raft_consensus.cc:729] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.441241 31464 consensus_queue.cc:260] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.441421 31461 leader_election.cc:288] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:23.441553 31464 ts_tablet_manager.cc:1398] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:23.441936 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cc49f0227d491ba7a8df281856b8f8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:23.442025 29862 raft_consensus.cc:3037] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.442812 29862 raft_consensus.cc:2454] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.443101 29405 leader_election.cc:302] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:23.443137 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7cc49f0227d491ba7a8df281856b8f8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:23.443238 29601 raft_consensus.cc:3037] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.443292 31461 raft_consensus.cc:2781] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:23.443369 31461 raft_consensus.cc:686] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:23.443514 31461 consensus_queue.cc:237] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.444106 29601 raft_consensus.cc:2454] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.445490 29323 catalog_manager.cc:5484] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:23.472517 31463 raft_consensus.cc:482] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:23.472625 31463 raft_consensus.cc:504] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.473088 31463 leader_election.cc:288] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:23.473484 31463 raft_consensus.cc:482] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:23.473495 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7650009dd8954f11ab2ecafef765692c" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:23.473567 31463 raft_consensus.cc:504] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.473531 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7650009dd8954f11ab2ecafef765692c" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:23.473616 29601 raft_consensus.cc:2454] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:23.473685 29733 raft_consensus.cc:2454] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:23.473889 31463 leader_election.cc:288] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:23.474004 29797 leader_election.cc:302] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:23.474160 31463 raft_consensus.cc:2781] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:23.474203 31463 raft_consensus.cc:482] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:23.474238 31463 raft_consensus.cc:3037] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.474339 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c687dbe084f7469fa0167fc769b5de65" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:23.474462 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c687dbe084f7469fa0167fc769b5de65" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:23.474501 29470 raft_consensus.cc:2454] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:23.474574 29733 raft_consensus.cc:2454] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:23.474871 29799 leader_election.cc:302] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:23.475056 30830 raft_consensus.cc:2781] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:23.475129 30830 raft_consensus.cc:482] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:23.475167 30830 raft_consensus.cc:3037] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.475703 31463 raft_consensus.cc:504] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.476059 31463 leader_election.cc:288] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:23.476086 30830 raft_consensus.cc:504] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.476388 30830 leader_election.cc:288] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:23.476408 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7650009dd8954f11ab2ecafef765692c" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:23.476526 29601 raft_consensus.cc:3037] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.476737 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c687dbe084f7469fa0167fc769b5de65" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:23.476868 29470 raft_consensus.cc:3037] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.476876 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c687dbe084f7469fa0167fc769b5de65" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:23.476917 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7650009dd8954f11ab2ecafef765692c" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:23.477008 29733 raft_consensus.cc:3037] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.477530 29601 raft_consensus.cc:2454] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:23.477795 29470 raft_consensus.cc:2454] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:23.477795 29797 leader_election.cc:302] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:23.478006 30830 raft_consensus.cc:2781] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:23.478005 29733 raft_consensus.cc:2454] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:23.478085 30830 raft_consensus.cc:686] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:23.478163 29799 leader_election.cc:302] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:23.478226 30830 consensus_queue.cc:237] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.478386 31410 raft_consensus.cc:2781] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:23.478464 31410 raft_consensus.cc:686] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:23.477013 29732 raft_consensus.cc:3037] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.478603 31410 consensus_queue.cc:237] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.479578 29732 raft_consensus.cc:2454] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:23.480854 29323 catalog_manager.cc:5484] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:23.481081 29323 catalog_manager.cc:5484] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:00:23.613068 31480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:23.613330 31480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:23.616446 31480 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240502 14:00:23.693773 29470 raft_consensus.cc:1261] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.693933 29732 raft_consensus.cc:1261] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.694307 31463 consensus_queue.cc:1035] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.694870 29864 raft_consensus.cc:1261] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.695281 31410 consensus_queue.cc:1035] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.696262 29600 raft_consensus.cc:1261] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.699146 31407 consensus_queue.cc:1035] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.699663 29732 raft_consensus.cc:1261] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.700028 31410 consensus_queue.cc:1035] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.700309 30830 consensus_queue.cc:1035] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:23.700891 29601 raft_consensus.cc:1261] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:23.701936 31461 consensus_queue.cc:1035] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.801 ms
time per row: 0.0534003 ms
I20240502 14:00:23.767565 29325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:23.773208 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 7bad6684944b481989728378ab51d515 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.773226 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 7bad6684944b481989728378ab51d515 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.773555 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bad6684944b481989728378ab51d515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.773557 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bad6684944b481989728378ab51d515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.773682 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 7bad6684944b481989728378ab51d515 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.773957 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bad6684944b481989728378ab51d515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.775223 29711 tablet_service.cc:1467] Processing CreateTablet for tablet e227b307a0f44a088bf50899d59f2118 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.775489 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e227b307a0f44a088bf50899d59f2118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.775642 29713 tablet_service.cc:1467] Processing CreateTablet for tablet bda9fdd55bbb40fdb56f5fac6c5a497b (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.775841 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda9fdd55bbb40fdb56f5fac6c5a497b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.775841 31464 tablet_bootstrap.cc:492] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:23.776247 29844 tablet_service.cc:1467] Processing CreateTablet for tablet e227b307a0f44a088bf50899d59f2118 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.776450 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e227b307a0f44a088bf50899d59f2118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.776655 31464 tablet_bootstrap.cc:654] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.777159 31468 tablet_bootstrap.cc:492] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:23.777556 31465 tablet_bootstrap.cc:492] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:23.777709 31466 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:23.778013 31468 tablet_bootstrap.cc:654] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.778430 31465 tablet_bootstrap.cc:654] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.778473 31466 tablet_bootstrap.cc:654] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.778887 29451 tablet_service.cc:1467] Processing CreateTablet for tablet e227b307a0f44a088bf50899d59f2118 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.779125 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e227b307a0f44a088bf50899d59f2118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.779698 31465 tablet_bootstrap.cc:492] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:23.779788 31465 ts_tablet_manager.cc:1367] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:23.780289 31465 raft_consensus.cc:348] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.780398 31465 raft_consensus.cc:374] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.780401 31466 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:23.780428 31465 raft_consensus.cc:729] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.780462 31466 ts_tablet_manager.cc:1367] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:23.780555 31465 consensus_queue.cc:260] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.780865 31465 ts_tablet_manager.cc:1398] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:23.780858 31466 raft_consensus.cc:348] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.780937 31466 raft_consensus.cc:374] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.780957 31466 raft_consensus.cc:729] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.781057 31466 consensus_queue.cc:260] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.781294 31466 ts_tablet_manager.cc:1398] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:23.781447 31466 tablet_bootstrap.cc:492] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:23.778887 29450 tablet_service.cc:1467] Processing CreateTablet for tablet bda9fdd55bbb40fdb56f5fac6c5a497b (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.781857 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda9fdd55bbb40fdb56f5fac6c5a497b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.782269 31466 tablet_bootstrap.cc:654] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.784227 31466 tablet_bootstrap.cc:492] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:23.784313 31466 ts_tablet_manager.cc:1367] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:23.784747 31466 raft_consensus.cc:348] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.784852 31466 raft_consensus.cc:374] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.784883 31466 raft_consensus.cc:729] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.785373 31466 consensus_queue.cc:260] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.785518 29581 tablet_service.cc:1467] Processing CreateTablet for tablet bda9fdd55bbb40fdb56f5fac6c5a497b (DEFAULT_TABLE table=rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:23.785704 31464 tablet_bootstrap.cc:492] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:23.785791 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda9fdd55bbb40fdb56f5fac6c5a497b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.785794 31464 ts_tablet_manager.cc:1367] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 14:00:23.786280 31466 ts_tablet_manager.cc:1398] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:23.786266 31464 raft_consensus.cc:348] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.786389 31464 raft_consensus.cc:374] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.786422 31464 raft_consensus.cc:729] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.786558 31464 consensus_queue.cc:260] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.786785 31464 ts_tablet_manager.cc:1398] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:23.786967 31464 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:23.787724 31464 tablet_bootstrap.cc:654] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.788352 31465 tablet_bootstrap.cc:492] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:23.789294 31465 tablet_bootstrap.cc:654] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.790066 31468 tablet_bootstrap.cc:492] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:23.790154 31468 ts_tablet_manager.cc:1367] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240502 14:00:23.790629 31468 raft_consensus.cc:348] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.790764 31468 raft_consensus.cc:374] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.790781 31465 tablet_bootstrap.cc:492] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:23.790813 31468 raft_consensus.cc:729] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.790890 31465 ts_tablet_manager.cc:1367] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:23.790992 31468 consensus_queue.cc:260] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.791268 31468 ts_tablet_manager.cc:1398] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:23.791332 31465 raft_consensus.cc:348] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.791433 31465 raft_consensus.cc:374] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.791466 31465 raft_consensus.cc:729] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.791494 31468 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:23.791652 31465 consensus_queue.cc:260] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.791709 31464 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:23.791787 31464 ts_tablet_manager.cc:1367] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:23.792085 31465 ts_tablet_manager.cc:1398] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:23.792181 31464 raft_consensus.cc:348] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.792272 31464 raft_consensus.cc:374] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.792310 31464 raft_consensus.cc:729] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.792446 31464 consensus_queue.cc:260] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.794137 31464 ts_tablet_manager.cc:1398] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:23.794994 31468 tablet_bootstrap.cc:654] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.797410 31468 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:23.797478 31468 ts_tablet_manager.cc:1367] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:00:23.797806 31468 raft_consensus.cc:348] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.797917 31468 raft_consensus.cc:374] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.797964 31468 raft_consensus.cc:729] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.798080 31468 consensus_queue.cc:260] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:23.798343 31468 ts_tablet_manager.cc:1398] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:23.798570 31468 tablet_bootstrap.cc:492] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:23.799391 31468 tablet_bootstrap.cc:654] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:23.800418 31468 tablet_bootstrap.cc:492] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:23.800508 31468 ts_tablet_manager.cc:1367] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:23.800921 31468 raft_consensus.cc:348] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.801002 31468 raft_consensus.cc:374] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:23.801023 31468 raft_consensus.cc:729] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:23.801144 31468 consensus_queue.cc:260] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.801520 31468 ts_tablet_manager.cc:1398] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:23.924320 31461 raft_consensus.cc:482] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:23.924461 31461 raft_consensus.cc:504] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.924907 31461 leader_election.cc:288] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:23.925242 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad6684944b481989728378ab51d515" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:23.925242 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad6684944b481989728378ab51d515" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:23.925385 29864 raft_consensus.cc:2454] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.925396 29601 raft_consensus.cc:2454] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.925658 29404 leader_election.cc:302] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:23.925815 31461 raft_consensus.cc:2781] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:23.925877 31461 raft_consensus.cc:482] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:23.925916 31461 raft_consensus.cc:3037] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.926745 31461 raft_consensus.cc:504] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.927085 31461 leader_election.cc:288] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:23.927376 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad6684944b481989728378ab51d515" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:23.927426 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad6684944b481989728378ab51d515" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:23.927484 29601 raft_consensus.cc:3037] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.927552 29864 raft_consensus.cc:3037] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.928335 29864 raft_consensus.cc:2454] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.928342 29601 raft_consensus.cc:2454] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.928588 29404 leader_election.cc:302] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:23.928759 31461 raft_consensus.cc:2781] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:23.928840 31461 raft_consensus.cc:686] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:23.929020 31461 consensus_queue.cc:237] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:23.930966 29323 catalog_manager.cc:5484] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:23.994174 31461 raft_consensus.cc:482] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:23.994326 31461 raft_consensus.cc:504] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.994676 31461 leader_election.cc:288] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:23.995040 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda9fdd55bbb40fdb56f5fac6c5a497b" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:23.995039 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda9fdd55bbb40fdb56f5fac6c5a497b" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:23.995177 29733 raft_consensus.cc:2454] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.995182 29601 raft_consensus.cc:2454] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:23.995445 29404 leader_election.cc:302] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:23.995616 31461 raft_consensus.cc:2781] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:23.995671 31461 raft_consensus.cc:482] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:23.995723 31461 raft_consensus.cc:3037] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.996534 31461 raft_consensus.cc:504] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:23.996857 31461 leader_election.cc:288] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:23.997167 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda9fdd55bbb40fdb56f5fac6c5a497b" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:23.997174 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda9fdd55bbb40fdb56f5fac6c5a497b" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:23.997294 29601 raft_consensus.cc:3037] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.997301 29733 raft_consensus.cc:3037] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:23.998073 29733 raft_consensus.cc:2454] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.998078 29601 raft_consensus.cc:2454] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:23.998347 29404 leader_election.cc:302] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:23.998560 31461 raft_consensus.cc:2781] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:23.998636 31461 raft_consensus.cc:686] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:23.998775 31461 consensus_queue.cc:237] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.000630 29323 catalog_manager.cc:5484] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:24.064971 31410 raft_consensus.cc:482] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:24.065145 31410 raft_consensus.cc:504] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.065613 31410 leader_election.cc:288] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:24.066074 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227b307a0f44a088bf50899d59f2118" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:24.066097 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227b307a0f44a088bf50899d59f2118" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:24.066232 29732 raft_consensus.cc:2454] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:24.066242 29470 raft_consensus.cc:2454] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:24.066565 29798 leader_election.cc:302] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.066769 31410 raft_consensus.cc:2781] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:24.066884 31410 raft_consensus.cc:482] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:24.066946 31410 raft_consensus.cc:3037] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.068025 31410 raft_consensus.cc:504] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.068367 31410 leader_election.cc:288] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:24.068670 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227b307a0f44a088bf50899d59f2118" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:24.068683 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227b307a0f44a088bf50899d59f2118" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:24.068773 29732 raft_consensus.cc:3037] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.068825 29470 raft_consensus.cc:3037] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.069597 29732 raft_consensus.cc:2454] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:24.069682 29470 raft_consensus.cc:2454] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:24.069872 29798 leader_election.cc:302] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.070034 31463 raft_consensus.cc:2781] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:24.070106 31463 raft_consensus.cc:686] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:24.070247 31463 consensus_queue.cc:237] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.072012 29323 catalog_manager.cc:5484] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:24.257427 31534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:24.257696 31534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:24.260886 31534 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240502 14:00:24.349505 29732 raft_consensus.cc:1261] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.349969 31463 consensus_queue.cc:1035] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.351905 29470 raft_consensus.cc:1261] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.352264 31463 consensus_queue.cc:1035] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.353981 29864 raft_consensus.cc:1261] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.354372 31461 consensus_queue.cc:1035] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.356349 29601 raft_consensus.cc:1261] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.356822 31461 consensus_queue.cc:1035] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.357695 29600 raft_consensus.cc:1261] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.359041 29732 raft_consensus.cc:1261] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.359455 31461 consensus_queue.cc:1035] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.360246 31461 consensus_queue.cc:1035] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.651 ms
time per row: 0.0623255 ms
I20240502 14:00:24.427457 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:24.433247 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 209e6922777b49ddb87a3dadf65d651a (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.433442 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 09ce1f1d33dd4beca86a484a6f498175 (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.433578 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209e6922777b49ddb87a3dadf65d651a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.433686 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ce1f1d33dd4beca86a484a6f498175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.434376 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 835183257412496b90defa7ecaa0ff13 (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.434609 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835183257412496b90defa7ecaa0ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.436064 31594 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:24.436100 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 209e6922777b49ddb87a3dadf65d651a (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.436322 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209e6922777b49ddb87a3dadf65d651a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.436717 31595 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:24.437126 31594 tablet_bootstrap.cc:654] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.437731 31595 tablet_bootstrap.cc:654] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.437778 31593 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:24.438481 31598 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:24.438668 31594 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:24.438670 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 209e6922777b49ddb87a3dadf65d651a (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.438735 31593 tablet_bootstrap.cc:654] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.438747 31594 ts_tablet_manager.cc:1367] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:24.438923 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209e6922777b49ddb87a3dadf65d651a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.439257 31594 raft_consensus.cc:348] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.439356 31594 raft_consensus.cc:374] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.439393 31594 raft_consensus.cc:729] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.439467 31598 tablet_bootstrap.cc:654] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.439507 31594 consensus_queue.cc:260] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.439728 31593 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:24.439782 31594 ts_tablet_manager.cc:1398] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:24.439801 31593 ts_tablet_manager.cc:1367] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:24.440296 31593 raft_consensus.cc:348] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.440399 31593 raft_consensus.cc:374] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.440431 31593 raft_consensus.cc:729] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.440573 31593 consensus_queue.cc:260] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.440883 31598 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:24.440899 31593 ts_tablet_manager.cc:1398] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.440966 31598 ts_tablet_manager.cc:1367] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:24.441130 31593 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:24.441082 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 09ce1f1d33dd4beca86a484a6f498175 (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.441330 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ce1f1d33dd4beca86a484a6f498175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.441444 31598 raft_consensus.cc:348] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.441553 31598 raft_consensus.cc:374] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.441589 31598 raft_consensus.cc:729] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.441686 31598 consensus_queue.cc:260] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.442026 31598 ts_tablet_manager.cc:1398] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:24.442277 31593 tablet_bootstrap.cc:654] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.443411 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 835183257412496b90defa7ecaa0ff13 (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.443641 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835183257412496b90defa7ecaa0ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.445341 31593 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:24.445416 31593 ts_tablet_manager.cc:1367] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:24.446691 31594 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:24.446677 31593 raft_consensus.cc:348] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.446780 31593 raft_consensus.cc:374] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.446820 31593 raft_consensus.cc:729] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.446962 31593 consensus_queue.cc:260] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.447201 31593 ts_tablet_manager.cc:1398] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:24.447626 31594 tablet_bootstrap.cc:654] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.448767 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 835183257412496b90defa7ecaa0ff13 (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.449048 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835183257412496b90defa7ecaa0ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.449079 31594 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:24.449172 31594 ts_tablet_manager.cc:1367] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:24.449563 31594 raft_consensus.cc:348] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.449662 31594 raft_consensus.cc:374] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.449704 31594 raft_consensus.cc:729] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.449831 31594 consensus_queue.cc:260] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.450242 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 09ce1f1d33dd4beca86a484a6f498175 (DEFAULT_TABLE table=rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.450505 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ce1f1d33dd4beca86a484a6f498175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.450959 31598 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:24.451696 31598 tablet_bootstrap.cc:654] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.452652 31598 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:24.452739 31598 ts_tablet_manager.cc:1367] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:24.453063 31594 ts_tablet_manager.cc:1398] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240502 14:00:24.453116 31598 raft_consensus.cc:348] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.453198 31598 raft_consensus.cc:374] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.453219 31598 raft_consensus.cc:729] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.453315 31598 consensus_queue.cc:260] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.453586 31598 ts_tablet_manager.cc:1398] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:24.453645 31594 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:24.453792 31598 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:24.454512 31594 tablet_bootstrap.cc:654] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.454624 31598 tablet_bootstrap.cc:654] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.455211 31373 raft_consensus.cc:482] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:24.455300 31373 raft_consensus.cc:504] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.455672 31595 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:24.455742 31373 leader_election.cc:288] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:24.455768 31595 ts_tablet_manager.cc:1367] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:00:24.455920 31598 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:24.455981 31598 ts_tablet_manager.cc:1367] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:24.456319 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ce1f1d33dd4beca86a484a6f498175" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:24.456372 31598 raft_consensus.cc:348] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.456463 31598 raft_consensus.cc:374] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.456494 31598 raft_consensus.cc:729] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.456475 29601 raft_consensus.cc:2454] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:24.456593 31598 consensus_queue.cc:260] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.456632 31595 raft_consensus.cc:348] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.456763 31595 raft_consensus.cc:374] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.456815 31595 raft_consensus.cc:729] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.456854 29666 leader_election.cc:302] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.456887 31598 ts_tablet_manager.cc:1398] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.457012 31373 raft_consensus.cc:2781] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:24.457067 31373 raft_consensus.cc:482] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:24.457109 31373 raft_consensus.cc:3037] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.457199 31595 consensus_queue.cc:260] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.457712 31595 ts_tablet_manager.cc:1398] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:24.458034 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ce1f1d33dd4beca86a484a6f498175" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:24.458168 29864 raft_consensus.cc:2454] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:24.458137 31373 raft_consensus.cc:504] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.458482 31373 leader_election.cc:288] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:24.458945 31594 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:24.459033 31594 ts_tablet_manager.cc:1367] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:24.459498 31594 raft_consensus.cc:348] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.459592 31594 raft_consensus.cc:374] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.459625 31594 raft_consensus.cc:729] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.459753 31594 consensus_queue.cc:260] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.460024 31594 ts_tablet_manager.cc:1398] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.460211 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ce1f1d33dd4beca86a484a6f498175" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:24.460299 29864 raft_consensus.cc:3037] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.461315 29864 raft_consensus.cc:2454] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:24.461321 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ce1f1d33dd4beca86a484a6f498175" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:24.461408 29601 raft_consensus.cc:3037] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.461596 29667 leader_election.cc:302] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.462357 29601 raft_consensus.cc:2454] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:24.463570 31373 raft_consensus.cc:2781] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:24.463790 31373 raft_consensus.cc:686] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:24.463943 31373 consensus_queue.cc:237] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.465898 29326 catalog_manager.cc:5484] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:24.507442 31131 raft_consensus.cc:482] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:24.507582 31131 raft_consensus.cc:504] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.508000 31131 leader_election.cc:288] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:24.508461 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835183257412496b90defa7ecaa0ff13" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:24.508616 29471 raft_consensus.cc:2454] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:24.508872 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835183257412496b90defa7ecaa0ff13" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:24.508936 29537 leader_election.cc:302] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:24.509007 29864 raft_consensus.cc:2454] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:24.509171 31131 raft_consensus.cc:2781] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:24.509231 31131 raft_consensus.cc:482] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:24.509279 31131 raft_consensus.cc:3037] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.510152 31131 raft_consensus.cc:504] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.510478 31131 leader_election.cc:288] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:24.510798 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835183257412496b90defa7ecaa0ff13" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:24.510807 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835183257412496b90defa7ecaa0ff13" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:24.510965 29864 raft_consensus.cc:3037] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.510972 29471 raft_consensus.cc:3037] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.511816 29471 raft_consensus.cc:2454] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:24.511951 29864 raft_consensus.cc:2454] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:24.512187 29537 leader_election.cc:302] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:24.512344 31131 raft_consensus.cc:2781] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:24.512408 31131 raft_consensus.cc:686] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:24.512579 31131 consensus_queue.cc:237] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:24.515031 29325 catalog_manager.cc:5484] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:24.584547 31606 raft_consensus.cc:482] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:24.584702 31606 raft_consensus.cc:504] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.585112 31606 leader_election.cc:288] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:24.585584 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e6922777b49ddb87a3dadf65d651a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:24.585584 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e6922777b49ddb87a3dadf65d651a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:24.585718 29601 raft_consensus.cc:2454] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:24.585728 29864 raft_consensus.cc:2454] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:24.586050 29666 leader_election.cc:302] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.586256 31606 raft_consensus.cc:2781] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:24.586328 31606 raft_consensus.cc:482] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:24.586365 31606 raft_consensus.cc:3037] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.587255 31606 raft_consensus.cc:504] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.587575 31606 leader_election.cc:288] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:24.587931 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e6922777b49ddb87a3dadf65d651a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:24.587946 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209e6922777b49ddb87a3dadf65d651a" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:24.588059 29864 raft_consensus.cc:3037] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.588059 29601 raft_consensus.cc:3037] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.589030 29864 raft_consensus.cc:2454] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:24.589030 29601 raft_consensus.cc:2454] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:24.589351 29667 leader_election.cc:302] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.589534 31606 raft_consensus.cc:2781] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:24.589637 31606 raft_consensus.cc:686] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:24.589787 31606 consensus_queue.cc:237] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.591718 29325 catalog_manager.cc:5484] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:24.718940 31610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:24.719295 31610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:24.722657 31610 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 14:00:24.811878 29864 raft_consensus.cc:1261] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.812669 31373 consensus_queue.cc:1035] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.814255 29471 raft_consensus.cc:1261] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.814823 31435 consensus_queue.cc:1035] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.815829 29863 raft_consensus.cc:1261] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.822371 29600 raft_consensus.cc:1261] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.822791 31606 consensus_queue.cc:1035] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.823370 31435 consensus_queue.cc:1035] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.827181 29863 raft_consensus.cc:1261] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.827656 31373 consensus_queue.cc:1035] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.828204 29601 raft_consensus.cc:1261] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:24.828688 31373 consensus_queue.cc:1035] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:24.854996 31571 mvcc.cc:205] Tried to move back new op lower bound from 7023240908083941376 to 7023240906605977600. Current Snapshot: MvccSnapshot[applied={T|T < 7023240908083941376}]
I20240502 14:00:24.855024 31567 mvcc.cc:205] Tried to move back new op lower bound from 7023240908083941376 to 7023240906605977600. Current Snapshot: MvccSnapshot[applied={T|T < 7023240908083941376}]
INSERT report
rows total: 2000
time total: 121.68 ms
time per row: 0.0608402 ms
I20240502 14:00:24.887832 29329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:24.893627 29450 tablet_service.cc:1467] Processing CreateTablet for tablet e6f7a8f5b0b047208246d639ccf24447 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.893990 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f7a8f5b0b047208246d639ccf24447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.894133 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 8856fc8643184e73bf7156c1186b4b80 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.894145 29580 tablet_service.cc:1467] Processing CreateTablet for tablet e6f7a8f5b0b047208246d639ccf24447 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.894408 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f7a8f5b0b047208246d639ccf24447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.894418 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8856fc8643184e73bf7156c1186b4b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.895937 31595 tablet_bootstrap.cc:492] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:24.896487 31594 tablet_bootstrap.cc:492] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:24.896524 31593 tablet_bootstrap.cc:492] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:24.896665 29713 tablet_service.cc:1467] Processing CreateTablet for tablet e6f7a8f5b0b047208246d639ccf24447 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.896865 31595 tablet_bootstrap.cc:654] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.896884 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f7a8f5b0b047208246d639ccf24447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.897434 31593 tablet_bootstrap.cc:654] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.897434 31594 tablet_bootstrap.cc:654] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.899187 29711 tablet_service.cc:1467] Processing CreateTablet for tablet ba65f51fa3f64a479e1e62042c1ca64e (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.899442 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba65f51fa3f64a479e1e62042c1ca64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.899518 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 8856fc8643184e73bf7156c1186b4b80 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.899777 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8856fc8643184e73bf7156c1186b4b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.900255 31593 tablet_bootstrap.cc:492] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:24.900328 31593 ts_tablet_manager.cc:1367] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:24.900787 31593 raft_consensus.cc:348] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:24.900889 31593 raft_consensus.cc:374] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.900923 31593 raft_consensus.cc:729] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.901051 31593 consensus_queue.cc:260] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:24.901309 31593 ts_tablet_manager.cc:1398] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.901484 31593 tablet_bootstrap.cc:492] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:24.901852 31598 tablet_bootstrap.cc:492] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:24.902417 31593 tablet_bootstrap.cc:654] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.902519 29451 tablet_service.cc:1467] Processing CreateTablet for tablet ba65f51fa3f64a479e1e62042c1ca64e (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.902683 31598 tablet_bootstrap.cc:654] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.902944 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba65f51fa3f64a479e1e62042c1ca64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.903541 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 8856fc8643184e73bf7156c1186b4b80 (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.903751 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8856fc8643184e73bf7156c1186b4b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.903838 29580 tablet_service.cc:1467] Processing CreateTablet for tablet ba65f51fa3f64a479e1e62042c1ca64e (DEFAULT_TABLE table=rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:24.904107 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba65f51fa3f64a479e1e62042c1ca64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:24.906833 31595 tablet_bootstrap.cc:492] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:24.907084 31595 ts_tablet_manager.cc:1367] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 14:00:24.907626 31598 tablet_bootstrap.cc:492] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:24.907581 31595 raft_consensus.cc:348] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.907692 31598 ts_tablet_manager.cc:1367] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:24.907696 31595 raft_consensus.cc:374] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.907734 31595 raft_consensus.cc:729] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.907862 31595 consensus_queue.cc:260] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.908061 31598 raft_consensus.cc:348] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:24.908180 31595 ts_tablet_manager.cc:1398] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:24.908188 31598 raft_consensus.cc:374] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.908222 31598 raft_consensus.cc:729] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.908313 31595 tablet_bootstrap.cc:492] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:24.908370 31598 consensus_queue.cc:260] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:24.908522 31594 tablet_bootstrap.cc:492] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:24.908597 31594 ts_tablet_manager.cc:1367] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:24.908653 31598 ts_tablet_manager.cc:1398] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.909049 31594 raft_consensus.cc:348] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.909149 31594 raft_consensus.cc:374] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.909173 31595 tablet_bootstrap.cc:654] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.909180 31594 raft_consensus.cc:729] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.909291 31594 consensus_queue.cc:260] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.909497 31594 ts_tablet_manager.cc:1398] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.909646 31594 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:24.910913 31593 tablet_bootstrap.cc:492] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:24.910940 31594 tablet_bootstrap.cc:654] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.911015 31593 ts_tablet_manager.cc:1367] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:24.911355 31595 tablet_bootstrap.cc:492] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:24.911440 31595 ts_tablet_manager.cc:1367] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:24.911470 31593 raft_consensus.cc:348] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.911576 31593 raft_consensus.cc:374] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.911609 31593 raft_consensus.cc:729] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.911715 31593 consensus_queue.cc:260] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.911888 31595 raft_consensus.cc:348] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:24.911974 31595 raft_consensus.cc:374] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.911985 31593 ts_tablet_manager.cc:1398] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:24.912009 31595 raft_consensus.cc:729] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.912163 31593 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:24.912123 31595 consensus_queue.cc:260] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:24.912412 31595 ts_tablet_manager.cc:1398] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:24.912575 31595 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:24.913017 31593 tablet_bootstrap.cc:654] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.913437 31595 tablet_bootstrap.cc:654] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:24.914103 31594 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:24.914211 31594 ts_tablet_manager.cc:1367] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:00:24.914582 31595 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:24.914664 31595 ts_tablet_manager.cc:1367] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:24.914629 31594 raft_consensus.cc:348] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.914727 31594 raft_consensus.cc:374] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.914757 31594 raft_consensus.cc:729] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.914875 31594 consensus_queue.cc:260] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.915135 31594 ts_tablet_manager.cc:1398] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.915310 31595 raft_consensus.cc:348] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.915401 31595 raft_consensus.cc:374] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.915434 31595 raft_consensus.cc:729] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.915542 31595 consensus_queue.cc:260] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.915813 31595 ts_tablet_manager.cc:1398] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.918004 31593 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:24.918093 31593 ts_tablet_manager.cc:1367] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:24.918560 31593 raft_consensus.cc:348] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.918675 31593 raft_consensus.cc:374] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:24.918710 31593 raft_consensus.cc:729] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:24.918819 31593 consensus_queue.cc:260] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:24.919153 31593 ts_tablet_manager.cc:1398] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:24.938642 31606 raft_consensus.cc:482] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:24.938772 31606 raft_consensus.cc:504] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.939160 31606 leader_election.cc:288] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:24.939523 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f7a8f5b0b047208246d639ccf24447" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:24.939520 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f7a8f5b0b047208246d639ccf24447" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:24.939661 29601 raft_consensus.cc:2454] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:24.939661 29470 raft_consensus.cc:2454] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:24.939978 29668 leader_election.cc:302] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.940387 31606 raft_consensus.cc:2781] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:24.940452 31606 raft_consensus.cc:482] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:24.940496 31606 raft_consensus.cc:3037] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.941463 31606 raft_consensus.cc:504] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.941794 31606 leader_election.cc:288] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:24.942116 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f7a8f5b0b047208246d639ccf24447" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:24.942236 29601 raft_consensus.cc:3037] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.942251 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f7a8f5b0b047208246d639ccf24447" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:24.942360 29470 raft_consensus.cc:3037] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:24.943238 29601 raft_consensus.cc:2454] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:24.943271 29470 raft_consensus.cc:2454] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:24.943537 29666 leader_election.cc:302] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:24.943727 31606 raft_consensus.cc:2781] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:24.943796 31606 raft_consensus.cc:686] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:24.943908 31606 consensus_queue.cc:237] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:24.945974 29327 catalog_manager.cc:5484] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:25.072510 31435 raft_consensus.cc:482] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:25.072651 31435 raft_consensus.cc:504] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.073056 31435 leader_election.cc:288] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:25.073523 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65f51fa3f64a479e1e62042c1ca64e" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:25.073523 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65f51fa3f64a479e1e62042c1ca64e" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:25.073679 29733 raft_consensus.cc:2454] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:25.073679 29470 raft_consensus.cc:2454] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:25.073987 29536 leader_election.cc:302] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:25.074155 31435 raft_consensus.cc:2781] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:25.074213 31435 raft_consensus.cc:482] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:25.074239 31435 raft_consensus.cc:3037] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.075158 31435 raft_consensus.cc:504] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.075498 31435 leader_election.cc:288] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:25.075867 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65f51fa3f64a479e1e62042c1ca64e" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:25.075867 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65f51fa3f64a479e1e62042c1ca64e" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:25.075991 29470 raft_consensus.cc:3037] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.075991 29733 raft_consensus.cc:3037] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.076944 29733 raft_consensus.cc:2454] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:25.076944 29470 raft_consensus.cc:2454] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:25.077229 29537 leader_election.cc:302] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:25.077390 31435 raft_consensus.cc:2781] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:25.077479 31435 raft_consensus.cc:686] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:25.077661 31435 consensus_queue.cc:237] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.079775 29327 catalog_manager.cc:5484] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:25.158345 31463 raft_consensus.cc:482] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:25.158502 31463 raft_consensus.cc:504] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.159137 31463 leader_election.cc:288] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:25.159545 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856fc8643184e73bf7156c1186b4b80" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:25.159608 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856fc8643184e73bf7156c1186b4b80" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:25.159688 29470 raft_consensus.cc:2454] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:25.159788 29733 raft_consensus.cc:2454] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:25.160012 29799 leader_election.cc:302] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:25.160188 31463 raft_consensus.cc:2781] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:25.160257 31463 raft_consensus.cc:482] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:25.160315 31463 raft_consensus.cc:3037] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.161211 31463 raft_consensus.cc:504] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.161595 31463 leader_election.cc:288] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:25.161940 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856fc8643184e73bf7156c1186b4b80" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:25.161975 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856fc8643184e73bf7156c1186b4b80" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:25.162060 29470 raft_consensus.cc:3037] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.162098 29733 raft_consensus.cc:3037] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.163051 29470 raft_consensus.cc:2454] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:25.163055 29733 raft_consensus.cc:2454] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:25.163403 29798 leader_election.cc:302] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:25.163592 31463 raft_consensus.cc:2781] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:25.163688 31463 raft_consensus.cc:686] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:25.163843 31463 consensus_queue.cc:237] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.166013 29327 catalog_manager.cc:5484] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:00:25.327628 31661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:25.327962 31661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:00:25.331350 31606 consensus_queue.cc:1035] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:25.332073 31661 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:00:25.337728 31373 consensus_queue.cc:1035] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
I20240502 14:00:25.429585 29732 raft_consensus.cc:1261] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:25.429601 29471 raft_consensus.cc:1261] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:25.430014 31131 consensus_queue.cc:1035] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:25.430171 31435 consensus_queue.cc:1035] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:25.465549 29471 raft_consensus.cc:1261] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:25.465948 29732 raft_consensus.cc:1261] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:25.466393 31639 consensus_queue.cc:1035] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:25.467856 31639 consensus_queue.cc:1035] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 131.467 ms
time per row: 0.0657337 ms
I20240502 14:00:25.510124 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:25.516196 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 8e41bff5f1484f008851e76cc095eb91 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.516330 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 8e41bff5f1484f008851e76cc095eb91 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.516531 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e41bff5f1484f008851e76cc095eb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.516567 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e41bff5f1484f008851e76cc095eb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.516978 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 8e41bff5f1484f008851e76cc095eb91 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.516978 29711 tablet_service.cc:1467] Processing CreateTablet for tablet aeaf1db0298741fa813660a8e38d01f9 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.517244 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaf1db0298741fa813660a8e38d01f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.517244 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e41bff5f1484f008851e76cc095eb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.518939 29713 tablet_service.cc:1467] Processing CreateTablet for tablet be8afc0891fb43fa9ffc863ad2bc90d2 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.519176 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8afc0891fb43fa9ffc863ad2bc90d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.519572 31710 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:25.520229 29580 tablet_service.cc:1467] Processing CreateTablet for tablet be8afc0891fb43fa9ffc863ad2bc90d2 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.520473 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8afc0891fb43fa9ffc863ad2bc90d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.520519 29451 tablet_service.cc:1467] Processing CreateTablet for tablet aeaf1db0298741fa813660a8e38d01f9 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.520736 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaf1db0298741fa813660a8e38d01f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.520735 31710 tablet_bootstrap.cc:654] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.521756 29844 tablet_service.cc:1467] Processing CreateTablet for tablet aeaf1db0298741fa813660a8e38d01f9 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.521983 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaf1db0298741fa813660a8e38d01f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.522704 31713 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:25.523144 29842 tablet_service.cc:1467] Processing CreateTablet for tablet be8afc0891fb43fa9ffc863ad2bc90d2 (DEFAULT_TABLE table=rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:25.523380 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8afc0891fb43fa9ffc863ad2bc90d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:25.523667 31713 tablet_bootstrap.cc:654] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.524066 31710 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:25.524160 31710 ts_tablet_manager.cc:1367] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:25.524369 31712 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:25.524576 31710 raft_consensus.cc:348] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.524670 31710 raft_consensus.cc:374] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.524708 31710 raft_consensus.cc:729] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.524850 31710 consensus_queue.cc:260] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.525203 31710 ts_tablet_manager.cc:1398] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:25.525337 31712 tablet_bootstrap.cc:654] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.525511 31710 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:25.526348 31710 tablet_bootstrap.cc:654] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.526574 31712 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:25.526654 31712 ts_tablet_manager.cc:1367] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:25.527109 31712 raft_consensus.cc:348] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.527231 31712 raft_consensus.cc:374] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.527269 31712 raft_consensus.cc:729] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.527370 31713 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:25.527403 31712 consensus_queue.cc:260] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.527479 31713 ts_tablet_manager.cc:1367] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:25.527673 31711 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:25.527740 31712 ts_tablet_manager.cc:1398] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:25.527909 31712 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:25.528203 31713 raft_consensus.cc:348] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.528314 31713 raft_consensus.cc:374] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.528347 31713 raft_consensus.cc:729] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.528463 31713 consensus_queue.cc:260] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.528739 31713 ts_tablet_manager.cc:1398] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:25.528895 31713 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:25.529085 31712 tablet_bootstrap.cc:654] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.529247 31711 tablet_bootstrap.cc:654] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.529817 31713 tablet_bootstrap.cc:654] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.530395 31710 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:25.530421 31711 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:25.530483 31710 ts_tablet_manager.cc:1367] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:25.530611 31711 ts_tablet_manager.cc:1367] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:25.530992 31712 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:25.531078 31712 ts_tablet_manager.cc:1367] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:25.531070 31710 raft_consensus.cc:348] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.531179 31710 raft_consensus.cc:374] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.531216 31710 raft_consensus.cc:729] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.531347 31710 consensus_queue.cc:260] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.531416 31711 raft_consensus.cc:348] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.531533 31712 raft_consensus.cc:348] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.531651 31712 raft_consensus.cc:374] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.531693 31712 raft_consensus.cc:729] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.531620 31711 raft_consensus.cc:374] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.531807 31711 raft_consensus.cc:729] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.531821 31712 consensus_queue.cc:260] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.532019 31711 consensus_queue.cc:260] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.532114 31712 ts_tablet_manager.cc:1398] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:25.532253 31463 raft_consensus.cc:482] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:25.532322 31463 raft_consensus.cc:504] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.532698 31463 leader_election.cc:288] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:25.533174 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e41bff5f1484f008851e76cc095eb91" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:25.533304 29600 raft_consensus.cc:2454] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:25.533516 31710 ts_tablet_manager.cc:1398] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:25.534531 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e41bff5f1484f008851e76cc095eb91" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:25.534663 29470 raft_consensus.cc:2454] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:25.534819 31711 ts_tablet_manager.cc:1398] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:25.534969 29799 leader_election.cc:302] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:25.535030 31711 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:25.535113 31463 raft_consensus.cc:2781] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:25.535157 31463 raft_consensus.cc:482] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:25.535204 31463 raft_consensus.cc:3037] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.536161 31463 raft_consensus.cc:504] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.536484 31463 leader_election.cc:288] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:25.537285 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e41bff5f1484f008851e76cc095eb91" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:25.537392 29471 raft_consensus.cc:3037] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.537971 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e41bff5f1484f008851e76cc095eb91" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:25.538117 29600 raft_consensus.cc:3037] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.538442 29471 raft_consensus.cc:2454] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:25.539502 29600 raft_consensus.cc:2454] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:25.539885 29799 leader_election.cc:302] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:25.540051 31463 raft_consensus.cc:2781] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:25.540138 31463 raft_consensus.cc:686] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:25.540251 31463 consensus_queue.cc:237] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:25.541038 31711 tablet_bootstrap.cc:654] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.541729 31713 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:25.541850 31713 ts_tablet_manager.cc:1367] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:00:25.542316 31713 raft_consensus.cc:348] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.542413 31713 raft_consensus.cc:374] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.542445 31713 raft_consensus.cc:729] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.542428 29321 catalog_manager.cc:5484] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:25.542572 31713 consensus_queue.cc:260] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.542909 31713 ts_tablet_manager.cc:1398] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:25.543303 31711 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:25.543404 31711 ts_tablet_manager.cc:1367] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:00:25.543820 31711 raft_consensus.cc:348] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.543910 31711 raft_consensus.cc:374] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.543951 31711 raft_consensus.cc:729] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.544068 31711 consensus_queue.cc:260] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.544376 31711 ts_tablet_manager.cc:1398] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:25.544550 31711 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:25.545292 31711 tablet_bootstrap.cc:654] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:25.546224 31711 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:25.546303 31711 ts_tablet_manager.cc:1367] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:25.546689 31711 raft_consensus.cc:348] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.546790 31711 raft_consensus.cc:374] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:25.546837 31711 raft_consensus.cc:729] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:25.546994 31711 consensus_queue.cc:260] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.547295 31711 ts_tablet_manager.cc:1398] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:25.710124 31463 raft_consensus.cc:482] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:25.710263 31463 raft_consensus.cc:504] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.710708 31463 leader_election.cc:288] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:25.710999 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:25.711135 29601 raft_consensus.cc:2454] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:25.711091 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:25.711243 29733 raft_consensus.cc:2454] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:25.711509 29797 leader_election.cc:302] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:25.711684 31463 raft_consensus.cc:2781] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:25.711751 31463 raft_consensus.cc:482] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:25.711797 31463 raft_consensus.cc:3037] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.712877 31463 raft_consensus.cc:504] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.713248 31463 leader_election.cc:288] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:25.713624 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:25.713647 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:25.713732 29733 raft_consensus.cc:3037] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.713761 29601 raft_consensus.cc:3037] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.714676 29733 raft_consensus.cc:2454] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:25.714810 29601 raft_consensus.cc:2454] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:25.714964 29798 leader_election.cc:302] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:25.715107 31463 raft_consensus.cc:2781] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:25.715256 31463 raft_consensus.cc:686] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:25.715385 31463 consensus_queue.cc:237] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:25.717512 29326 catalog_manager.cc:5484] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:25.803519 31578 raft_consensus.cc:482] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:25.803647 31578 raft_consensus.cc:504] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.804136 31578 leader_election.cc:288] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:25.804694 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaf1db0298741fa813660a8e38d01f9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:25.804790 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaf1db0298741fa813660a8e38d01f9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:25.804855 29733 raft_consensus.cc:2454] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:25.804939 29863 raft_consensus.cc:2454] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:25.805155 29405 leader_election.cc:302] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:25.805349 31578 raft_consensus.cc:2781] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:25.805409 31578 raft_consensus.cc:482] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:25.805464 31578 raft_consensus.cc:3037] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.806355 31578 raft_consensus.cc:504] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.806684 31578 leader_election.cc:288] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:25.807021 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaf1db0298741fa813660a8e38d01f9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:25.807160 29732 raft_consensus.cc:3037] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.807242 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaf1db0298741fa813660a8e38d01f9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:25.807368 29863 raft_consensus.cc:3037] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:25.808040 29732 raft_consensus.cc:2454] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:25.808089 29863 raft_consensus.cc:2454] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:25.808337 29405 leader_election.cc:302] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:25.808553 31578 raft_consensus.cc:2781] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:25.808677 31578 raft_consensus.cc:686] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:25.808835 31578 consensus_queue.cc:237] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:25.811286 29326 catalog_manager.cc:5484] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:00:25.941871 31726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:25.942160 31726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:25.945441 31726 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240502 14:00:25.993316 31463 consensus_queue.cc:1035] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:26.004876 31707 consensus_queue.cc:1035] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:26.041905 29732 raft_consensus.cc:1261] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:26.042444 31407 consensus_queue.cc:1035] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:26.055091 29863 raft_consensus.cc:1261] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:26.057026 31407 consensus_queue.cc:1035] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:26.077132 29601 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:26.077167 29733 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:26.077638 31463 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:26.078686 31463 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 130.283 ms
time per row: 0.0651415 ms
I20240502 14:00:26.126655 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.133:39647
--local_ip_for_outbound_sockets=127.22.55.133
--tserver_master_addrs=127.22.55.190:32851
--webserver_port=37223
--webserver_interface=127.22.55.133
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:26.249375 31779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:26.249601 31779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:26.249650 31779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:26.253108 31779 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:26.253178 31779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:26.253249 31779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.133
I20240502 14:00:26.256914 31779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.133:39647
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.55.133
--webserver_port=37223
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.31779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.133
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:26.257788 31779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:26.258841 31779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:00:26.268177 31779 server_base.cc:1034] running on GCE node
W20240502 14:00:26.268419 31787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.268849 31784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.269966 31785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:26.270323 31779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:26.270887 31779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:26.272038 31779 hybrid_clock.cc:648] HybridClock initialized: now 1714658426272025 us; error 29 us; skew 500 ppm
I20240502 14:00:26.273916 31779 webserver.cc:460] Webserver started at http://127.22.55.133:37223/ using document root <none> and password file <none>
I20240502 14:00:26.274458 31779 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:26.274530 31779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:26.278126 31779 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:26.280213 31793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.281224 31779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:26.281343 31779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:26.281826 31779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:26.309701 31779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:26.310513 31779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:26.310674 31779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:26.311331 31779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:26.312333 31779 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:26.312556 31779 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.312619 31779 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:26.312639 31779 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.322551 31779 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.133:39647
I20240502 14:00:26.322588 31906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.133:39647 every 8 connection(s)
I20240502 14:00:26.323807 31779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:26.333331 31907 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:26.333559 31907 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:26.333873 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 31779
I20240502 14:00:26.334007 31907 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:26.334383 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.134:44137
--local_ip_for_outbound_sockets=127.22.55.134
--tserver_master_addrs=127.22.55.190:32851
--webserver_port=45535
--webserver_interface=127.22.55.134
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:26.335392 29326 ts_manager.cc:194] Re-registered known tserver with Master: f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:26.336208 29326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.133:44357
W20240502 14:00:26.451965 31910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:26.452183 31910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:26.452241 31910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:26.455652 31910 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:26.455718 31910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:26.455801 31910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.134
I20240502 14:00:26.459404 31910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.134:44137
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.55.134
--webserver_port=45535
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.31910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.134
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:26.460376 31910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:26.461467 31910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:26.469909 31916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.470630 31915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.471436 31918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:26.472308 31910 server_base.cc:1034] running on GCE node
I20240502 14:00:26.472709 31910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:26.473363 31910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:26.474520 31910 hybrid_clock.cc:648] HybridClock initialized: now 1714658426474497 us; error 34 us; skew 500 ppm
I20240502 14:00:26.476553 31910 webserver.cc:460] Webserver started at http://127.22.55.134:45535/ using document root <none> and password file <none>
I20240502 14:00:26.477079 31910 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:26.477160 31910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:26.480574 31910 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20240502 14:00:26.482774 31924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.483937 31910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:26.484052 31910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:26.484540 31910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:26.505270 31910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:26.506013 31910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:26.506179 31910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:26.506769 31910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:26.507826 31910 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:26.507880 31910 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.507930 31910 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:26.507958 31910 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.517822 31910 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.134:44137
I20240502 14:00:26.517853 32037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.134:44137 every 8 connection(s)
I20240502 14:00:26.519026 31910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:00:26.521126 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 31910
I20240502 14:00:26.521584 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.135:40995
--local_ip_for_outbound_sockets=127.22.55.135
--tserver_master_addrs=127.22.55.190:32851
--webserver_port=39467
--webserver_interface=127.22.55.135
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:26.529937 32038 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:26.530207 32038 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:26.530807 32038 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:26.531916 29326 ts_manager.cc:194] Re-registered known tserver with Master: 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:26.532487 29326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.134:44381
W20240502 14:00:26.635494 32041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:26.635810 32041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:26.635896 32041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:26.639611 32041 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:26.639679 32041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:26.639775 32041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.135
I20240502 14:00:26.643556 32041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.135:40995
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.55.135
--webserver_port=39467
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.32041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.135
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:26.644470 32041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:26.645622 32041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:26.654402 32046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.655012 32049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.655591 32047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:26.656481 32041 server_base.cc:1034] running on GCE node
I20240502 14:00:26.656858 32041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:26.657423 32041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:26.658576 32041 hybrid_clock.cc:648] HybridClock initialized: now 1714658426658560 us; error 28 us; skew 500 ppm
I20240502 14:00:26.660602 32041 webserver.cc:460] Webserver started at http://127.22.55.135:39467/ using document root <none> and password file <none>
I20240502 14:00:26.661264 32041 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:26.661336 32041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:26.664767 32041 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:00:26.666543 32055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.667805 32041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:00:26.667906 32041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
format_stamp: "Formatted at 2024-05-02 14:00:10 on dist-test-slave-sdf5"
I20240502 14:00:26.668304 32041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:26.682652 32041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:26.683352 32041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:26.683491 32041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:26.683956 32041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:26.684834 32041 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:26.684880 32041 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.684918 32041 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:26.684940 32041 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.694594 32041 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.135:40995
I20240502 14:00:26.694644 32168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.135:40995 every 8 connection(s)
I20240502 14:00:26.695719 32041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:00:26.698009 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 32041
I20240502 14:00:26.698454 22750 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCplTwR/build/debug/bin/kudu
/tmp/dist-test-taskCplTwR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.55.136:44201
--local_ip_for_outbound_sockets=127.22.55.136
--tserver_master_addrs=127.22.55.190:32851
--webserver_port=46663
--webserver_interface=127.22.55.136
--builtin_ntp_servers=127.22.55.148:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:26.706359 32169 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:26.706598 32169 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:26.707152 32169 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:26.708382 29326 ts_manager.cc:194] Re-registered known tserver with Master: f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:26.709061 29326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.135:54011
W20240502 14:00:26.817054 32172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:26.817301 32172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:26.817375 32172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:26.820703 32172 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:26.820791 32172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:26.820892 32172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.55.136
I20240502 14:00:26.824795 32172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.55.148:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.55.136:44201
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.55.136
--webserver_port=46663
--tserver_master_addrs=127.22.55.190:32851
--never_fsync=true
--heap_profile_path=/tmp/kudu.32172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.55.136
--log_dir=/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:26.825986 32172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:26.827134 32172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:26.835961 32178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.836692 32181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:26.837455 32179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:26.838266 32172 server_base.cc:1034] running on GCE node
I20240502 14:00:26.838644 32172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:26.839397 32172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:26.840566 32172 hybrid_clock.cc:648] HybridClock initialized: now 1714658426840427 us; error 157 us; skew 500 ppm
I20240502 14:00:26.842334 32172 webserver.cc:460] Webserver started at http://127.22.55.136:46663/ using document root <none> and password file <none>
I20240502 14:00:26.843009 32172 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:26.843112 32172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:26.846642 32172 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:26.848610 32187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.849763 32172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:26.849887 32172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8973f2527acb48f5ad36589652de73e1"
format_stamp: "Formatted at 2024-05-02 14:00:11 on dist-test-slave-sdf5"
I20240502 14:00:26.850407 32172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:26.866261 32172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:26.866935 32172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:26.867067 32172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:26.867568 32172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:26.868481 32172 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:26.868526 32172 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.868574 32172 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:26.868604 32172 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:26.877979 32172 rpc_server.cc:307] RPC server started. Bound to: 127.22.55.136:44201
I20240502 14:00:26.878031 32300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.55.136:44201 every 8 connection(s)
I20240502 14:00:26.879261 32172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:00:26.885524 22750 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCplTwR/build/debug/bin/kudu as pid 32172
I20240502 14:00:26.888669 32301 heartbeater.cc:347] Connected to a master server at 127.22.55.190:32851
I20240502 14:00:26.888940 32301 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:26.889541 32301 heartbeater.cc:511] Master 127.22.55.190:32851 requested a full tablet report, sending...
I20240502 14:00:26.890563 29324 ts_manager.cc:194] Re-registered known tserver with Master: 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:26.890475 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:26.891273 29324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.55.136:41895
I20240502 14:00:26.898988 29713 tablet_service.cc:1467] Processing CreateTablet for tablet ff33763a720b48edbbf5ddc96861a615 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9]), partition=
I20240502 14:00:26.899289 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff33763a720b48edbbf5ddc96861a615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:26.902314 32317 tablet_bootstrap.cc:492] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:26.903322 32317 tablet_bootstrap.cc:654] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:26.909715 32103 tablet_service.cc:1467] Processing CreateTablet for tablet ff33763a720b48edbbf5ddc96861a615 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9]), partition=
I20240502 14:00:26.910508 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff33763a720b48edbbf5ddc96861a615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:26.911195 31841 tablet_service.cc:1467] Processing CreateTablet for tablet ff33763a720b48edbbf5ddc96861a615 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9]), partition=
I20240502 14:00:26.911584 32317 tablet_bootstrap.cc:492] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:26.911680 32317 ts_tablet_manager.cc:1367] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:00:26.911782 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff33763a720b48edbbf5ddc96861a615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:26.912468 32317 raft_consensus.cc:348] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:26.912593 32317 raft_consensus.cc:374] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:26.912622 32317 raft_consensus.cc:729] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:26.912757 32317 consensus_queue.cc:260] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:26.913053 32317 ts_tablet_manager.cc:1398] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:26.918905 32319 tablet_bootstrap.cc:492] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:26.920953 32319 tablet_bootstrap.cc:654] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:26.921748 32319 log.cc:824] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:26.924412 32319 tablet_bootstrap.cc:492] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:26.924598 32319 ts_tablet_manager.cc:1367] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:00:26.924774 32321 tablet_bootstrap.cc:492] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:26.926924 32321 tablet_bootstrap.cc:654] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:26.927731 32321 log.cc:824] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:26.927836 32319 raft_consensus.cc:348] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:26.928604 32319 raft_consensus.cc:374] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:26.928659 32319 raft_consensus.cc:729] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:26.929113 32319 consensus_queue.cc:260] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:26.929600 32321 tablet_bootstrap.cc:492] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:26.929689 31907 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
I20240502 14:00:26.929759 32321 ts_tablet_manager.cc:1367] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:00:26.929916 32319 ts_tablet_manager.cc:1398] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20240502 14:00:26.933358 32321 raft_consensus.cc:348] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:26.933566 32321 raft_consensus.cc:374] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:26.933627 32321 raft_consensus.cc:729] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:26.934041 32321 consensus_queue.cc:260] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:26.934677 32169 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
I20240502 14:00:26.934907 32321 ts_tablet_manager.cc:1398] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20240502 14:00:26.946488 32170 tablet.cc:2369] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:27.013490 32324 raft_consensus.cc:482] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:27.013679 32324 raft_consensus.cc:504] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:27.015398 32324 leader_election.cc:288] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:27.019327 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff33763a720b48edbbf5ddc96861a615" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:27.019495 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff33763a720b48edbbf5ddc96861a615" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:27.019621 29733 raft_consensus.cc:2454] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:27.019627 31861 raft_consensus.cc:2454] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:27.019963 32057 leader_election.cc:302] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:27.020222 32324 raft_consensus.cc:2781] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:27.020294 32324 raft_consensus.cc:482] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:27.020344 32324 raft_consensus.cc:3037] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.021307 32324 raft_consensus.cc:504] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:27.021665 32324 leader_election.cc:288] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:27.022018 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff33763a720b48edbbf5ddc96861a615" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:27.022049 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff33763a720b48edbbf5ddc96861a615" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:27.022132 29733 raft_consensus.cc:3037] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.022157 31861 raft_consensus.cc:3037] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.023085 31861 raft_consensus.cc:2454] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:27.023186 29733 raft_consensus.cc:2454] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:27.023430 32057 leader_election.cc:302] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:27.023598 32324 raft_consensus.cc:2781] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:27.023770 32324 raft_consensus.cc:686] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:27.024068 32324 consensus_queue.cc:237] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:27.027017 29326 catalog_manager.cc:5484] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:00:27.075125 31908 tablet.cc:2369] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:27.132155 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:27.138389 29733 raft_consensus.cc:1261] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:27.138783 31861 raft_consensus.cc:1261] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:27.139207 32324 consensus_queue.cc:1035] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:27.139422 32330 consensus_queue.cc:1035] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:27.142700 29323 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet ff33763a720b48edbbf5ddc96861a615 (table rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9]). Expected version 1 got 0
I20240502 14:00:27.144039 29328 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet ff33763a720b48edbbf5ddc96861a615 (table rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9]). Expected version 1 got 0
I20240502 14:00:27.148329 29323 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet ff33763a720b48edbbf5ddc96861a615 (table rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9]). Expected version 1 got 0
I20240502 14:00:27.149822 32382 mvcc.cc:205] Tried to move back new op lower bound from 7023240917553348608 to 7023240917093163008. Current Snapshot: MvccSnapshot[applied={T|T < 7023240917553348608}]
I20240502 14:00:27.150347 29328 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet ff33763a720b48edbbf5ddc96861a615 (table rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9]). Expected version 1 got 0
I20240502 14:00:27.153791 32391 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:27.154702 32391 tablet.cc:1620] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:27.158174 29842 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:27.162261 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9] alter complete (version 1)
I20240502 14:00:27.163728 29451 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:27.163867 32392 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:27.164435 32393 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:27.165199 32393 tablet.cc:1620] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:27.167759 32392 tablet.cc:1620] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:27.169080 32103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:27.177265 29713 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:27.178366 31972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.178588 31841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.180871 32235 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.195528 29580 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:27.272825 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:27.279275 32391 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:27.279615 32391 tablet.cc:1620] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:27.279834 32392 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:27.280002 32393 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:27.280135 32392 tablet.cc:1620] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:27.280303 32393 tablet.cc:1620] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:27.285419 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0a940a9153844ba8ad7e8f05d3ff41b9] alter complete (version 2)
I20240502 14:00:27.364928 31972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.375567 29451 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:27.377210 29713 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:27.379912 32235 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.381116 29580 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:27.381865 29842 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:27.387234 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:27.389734 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:27.395650 32391 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:27.397342 32393 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:27.398689 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 19d48f827a6243448d344ee44a54482d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.399031 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d48f827a6243448d344ee44a54482d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.400619 32103 tablet_service.cc:1467] Processing CreateTablet for tablet e4a84488994c47ba9436ea7161d9b2e3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.400938 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a84488994c47ba9436ea7161d9b2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.402163 29320 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 14:00:27.402307 29320 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 14:00:27.398798 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 27e9d8ef4f754df3817cc4a981549e02 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.403450 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e9d8ef4f754df3817cc4a981549e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.403810 29842 tablet_service.cc:1467] Processing CreateTablet for tablet e4a84488994c47ba9436ea7161d9b2e3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.404086 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a84488994c47ba9436ea7161d9b2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.405321 32392 tablet.cc:1722] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:27.405699 32319 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:27.406600 32319 tablet_bootstrap.cc:654] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.408394 32452 tablet_bootstrap.cc:492] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:27.409308 32452 tablet_bootstrap.cc:654] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.398797 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 19d48f827a6243448d344ee44a54482d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.410259 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d48f827a6243448d344ee44a54482d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.413707 32319 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:27.413800 32319 ts_tablet_manager.cc:1367] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:00:27.414456 32319 raft_consensus.cc:348] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.414564 32319 raft_consensus.cc:374] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.414594 32319 raft_consensus.cc:729] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.414700 32319 consensus_queue.cc:260] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.415231 32319 ts_tablet_manager.cc:1398] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.415386 32319 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:27.416694 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 19d48f827a6243448d344ee44a54482d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.417157 32102 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:27.417658 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d48f827a6243448d344ee44a54482d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.418900 32234 tablet_service.cc:1467] Processing CreateTablet for tablet 27e9d8ef4f754df3817cc4a981549e02 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.419137 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e9d8ef4f754df3817cc4a981549e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.416602 32319 tablet_bootstrap.cc:654] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.427718 32319 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:27.427826 32319 ts_tablet_manager.cc:1367] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:27.428066 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 67a3a069498247aea44c04fe6ec1860d
I20240502 14:00:27.428293 32319 raft_consensus.cc:348] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.428398 32319 raft_consensus.cc:374] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.428428 32319 raft_consensus.cc:729] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.428445 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 5548ad03819446c88aaa61540ffb193d
I20240502 14:00:27.428536 32319 consensus_queue.cc:260] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.428655 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 7345a902a3754d4a98107d3a968f6eb1
I20240502 14:00:27.428826 32319 ts_tablet_manager.cc:1398] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.429025 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=0a940a9153844ba8ad7e8f05d3ff41b9] alter complete (version 3)
I20240502 14:00:27.430078 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 7345a902a3754d4a98107d3a968f6eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:27.430332 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 5548ad03819446c88aaa61540ffb193d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:27.430541 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a3a069498247aea44c04fe6ec1860d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:27.433624 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 27e9d8ef4f754df3817cc4a981549e02 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.434659 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e9d8ef4f754df3817cc4a981549e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.435354 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a3a069498247aea44c04fe6ec1860d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:27.435344 29711 tablet_service.cc:1514] Processing DeleteTablet for tablet 7345a902a3754d4a98107d3a968f6eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:27.433624 31971 tablet_service.cc:1467] Processing CreateTablet for tablet e4a84488994c47ba9436ea7161d9b2e3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.435528 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet 5548ad03819446c88aaa61540ffb193d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:27.436669 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a84488994c47ba9436ea7161d9b2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.436630 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet 5548ad03819446c88aaa61540ffb193d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:27.442679 32458 tablet_replica.cc:330] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:27.442943 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a3a069498247aea44c04fe6ec1860d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:27.443176 32458 raft_consensus.cc:2229] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.443202 32321 tablet_bootstrap.cc:492] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:27.443436 32458 raft_consensus.cc:2258] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.444315 32321 tablet_bootstrap.cc:654] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.444613 32458 ts_tablet_manager.cc:1875] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.445194 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 7345a902a3754d4a98107d3a968f6eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:27.447325 32458 ts_tablet_manager.cc:1888] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.447674 32458 log.cc:1196] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/5548ad03819446c88aaa61540ffb193d
I20240502 14:00:27.448117 32458 ts_tablet_manager.cc:1909] T 5548ad03819446c88aaa61540ffb193d P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:27.450142 32321 tablet_bootstrap.cc:492] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:27.450237 32321 ts_tablet_manager.cc:1367] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:27.450712 32457 tablet_replica.cc:330] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:27.450796 32321 raft_consensus.cc:348] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.450942 32321 raft_consensus.cc:374] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.450975 32321 raft_consensus.cc:729] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.451035 32457 raft_consensus.cc:2229] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.451126 32321 consensus_queue.cc:260] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.451412 32457 raft_consensus.cc:2258] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 14:00:27.453200 29668 consensus_peers.cc:485] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:27.453475 32321 ts_tablet_manager.cc:1398] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:27.455487 32463 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:27.456998 32462 tablet_bootstrap.cc:492] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:27.459177 31841 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.459582 32460 tablet_replica.cc:330] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:27.459903 32460 raft_consensus.cc:2229] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:27.460243 32460 raft_consensus.cc:2258] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.461274 32460 ts_tablet_manager.cc:1875] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.461623 32457 ts_tablet_manager.cc:1875] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.463212 32459 tablet_replica.cc:330] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:27.463496 32459 raft_consensus.cc:2229] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:27.463683 32460 ts_tablet_manager.cc:1888] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.463757 32460 log.cc:1196] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/5548ad03819446c88aaa61540ffb193d
I20240502 14:00:27.463874 32459 raft_consensus.cc:2258] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.464040 32460 ts_tablet_manager.cc:1909] T 5548ad03819446c88aaa61540ffb193d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:27.464779 32459 ts_tablet_manager.cc:1875] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.465618 32451 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:27.466977 32459 ts_tablet_manager.cc:1888] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.466981 32463 tablet_bootstrap.cc:654] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.467062 32459 log.cc:1196] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/7345a902a3754d4a98107d3a968f6eb1
I20240502 14:00:27.467367 32459 ts_tablet_manager.cc:1909] T 7345a902a3754d4a98107d3a968f6eb1 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:27.468322 32459 tablet_replica.cc:330] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:27.468456 32459 raft_consensus.cc:2229] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.468441 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 7345a902a3754d4a98107d3a968f6eb1 (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.468596 32459 raft_consensus.cc:2258] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.465620 32457 ts_tablet_manager.cc:1888] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.468850 32457 log.cc:1196] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/7345a902a3754d4a98107d3a968f6eb1
I20240502 14:00:27.469157 32459 ts_tablet_manager.cc:1875] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.469326 32457 ts_tablet_manager.cc:1909] T 7345a902a3754d4a98107d3a968f6eb1 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:27.469425 32462 tablet_bootstrap.cc:654] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.472967 32457 tablet_replica.cc:330] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:27.473022 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 7345a902a3754d4a98107d3a968f6eb1 (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.473222 32457 raft_consensus.cc:2229] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:27.474212 32462 log.cc:824] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:27.474265 32452 tablet_bootstrap.cc:492] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:27.474371 32452 ts_tablet_manager.cc:1367] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.000s
I20240502 14:00:27.474835 32452 raft_consensus.cc:348] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.474994 32452 raft_consensus.cc:374] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.475021 32452 raft_consensus.cc:729] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.475154 32452 consensus_queue.cc:260] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.475412 32452 ts_tablet_manager.cc:1398] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:27.476727 32451 tablet_bootstrap.cc:654] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.467832 32463 log.cc:824] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:27.476181 32457 raft_consensus.cc:2258] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.478238 32457 ts_tablet_manager.cc:1875] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.478874 32462 tablet_bootstrap.cc:492] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:27.479089 32462 ts_tablet_manager.cc:1367] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 14:00:27.480135 32459 ts_tablet_manager.cc:1888] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.480283 32459 log.cc:1196] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/67a3a069498247aea44c04fe6ec1860d
I20240502 14:00:27.480692 32459 ts_tablet_manager.cc:1909] T 67a3a069498247aea44c04fe6ec1860d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:27.481642 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 67a3a069498247aea44c04fe6ec1860d (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.482642 32457 ts_tablet_manager.cc:1888] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.483464 32457 log.cc:1196] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/67a3a069498247aea44c04fe6ec1860d
I20240502 14:00:27.483772 32457 ts_tablet_manager.cc:1909] T 67a3a069498247aea44c04fe6ec1860d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:27.484535 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 67a3a069498247aea44c04fe6ec1860d (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.484807 32457 tablet_replica.cc:330] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:27.484963 32457 raft_consensus.cc:2229] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.485186 32460 tablet_replica.cc:330] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:27.485260 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 5548ad03819446c88aaa61540ffb193d (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.485337 32460 raft_consensus.cc:2229] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.485430 32457 raft_consensus.cc:2258] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.485462 32460 raft_consensus.cc:2258] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.485980 32457 ts_tablet_manager.cc:1875] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.486675 32460 ts_tablet_manager.cc:1875] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.488466 32457 ts_tablet_manager.cc:1888] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.488551 32457 log.cc:1196] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/5548ad03819446c88aaa61540ffb193d
I20240502 14:00:27.489091 32457 ts_tablet_manager.cc:1909] T 5548ad03819446c88aaa61540ffb193d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:27.490090 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 5548ad03819446c88aaa61540ffb193d (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.491793 32460 ts_tablet_manager.cc:1888] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.491868 32460 log.cc:1196] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/7345a902a3754d4a98107d3a968f6eb1
I20240502 14:00:27.492130 32460 ts_tablet_manager.cc:1909] T 7345a902a3754d4a98107d3a968f6eb1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:27.492692 32462 raft_consensus.cc:348] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.492973 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 7345a902a3754d4a98107d3a968f6eb1 (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.493036 32462 raft_consensus.cc:374] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.493199 32462 raft_consensus.cc:729] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.493729 32462 consensus_queue.cc:260] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.494681 32038 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
I20240502 14:00:27.495860 32462 ts_tablet_manager.cc:1398] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240502 14:00:27.496073 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 5548ad03819446c88aaa61540ffb193d (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.496093 32458 tablet_replica.cc:330] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:27.496266 32458 raft_consensus.cc:2229] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.496438 32458 raft_consensus.cc:2258] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.496444 32462 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:27.498134 32462 tablet_bootstrap.cc:654] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.499491 32462 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:27.499576 32462 ts_tablet_manager.cc:1367] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:00:27.499619 32458 ts_tablet_manager.cc:1875] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.500263 32462 raft_consensus.cc:348] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.500391 32462 raft_consensus.cc:374] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.500440 32462 raft_consensus.cc:729] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.500566 32462 consensus_queue.cc:260] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.500823 32462 ts_tablet_manager.cc:1398] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.502066 32458 ts_tablet_manager.cc:1888] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.502229 32458 log.cc:1196] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/67a3a069498247aea44c04fe6ec1860d
I20240502 14:00:27.502558 32458 ts_tablet_manager.cc:1909] T 67a3a069498247aea44c04fe6ec1860d P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:27.503487 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 67a3a069498247aea44c04fe6ec1860d (table rebalance_test_table_0 [id=360810cd761a40079b8e76f454ab28fc]) successfully deleted
I20240502 14:00:27.506893 32451 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:27.506994 32451 ts_tablet_manager.cc:1367] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240502 14:00:27.507416 32451 raft_consensus.cc:348] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.507519 32451 raft_consensus.cc:374] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.507547 32451 raft_consensus.cc:729] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.507658 32451 consensus_queue.cc:260] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.507933 32451 ts_tablet_manager.cc:1398] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:27.509380 32463 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:27.509590 32463 ts_tablet_manager.cc:1367] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.002s
I20240502 14:00:27.514106 32463 raft_consensus.cc:348] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.514304 32463 raft_consensus.cc:374] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.514361 32463 raft_consensus.cc:729] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.514783 32463 consensus_queue.cc:260] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.515759 32463 ts_tablet_manager.cc:1398] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 14:00:27.516149 32301 heartbeater.cc:503] Master 127.22.55.190:32851 was elected leader, sending a full tablet report...
I20240502 14:00:27.517247 32463 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:27.518421 32463 tablet_bootstrap.cc:654] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
W20240502 14:00:27.520392 32039 tablet.cc:2369] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:27.527911 32463 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:27.528026 32463 ts_tablet_manager.cc:1367] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:27.528681 32463 raft_consensus.cc:348] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.528812 32463 raft_consensus.cc:374] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.528849 32463 raft_consensus.cc:729] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.529047 32463 consensus_queue.cc:260] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.529371 32463 ts_tablet_manager.cc:1398] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:27.539189 29326 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:27.539348 29326 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:27.541694 29326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet ff33763a720b48edbbf5ddc96861a615
I20240502 14:00:27.542515 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet ff33763a720b48edbbf5ddc96861a615 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:27.542546 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet ff33763a720b48edbbf5ddc96861a615 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:27.542547 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet ff33763a720b48edbbf5ddc96861a615 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:27.542690 32459 tablet_replica.cc:330] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:27.542817 32459 raft_consensus.cc:2229] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.542963 32459 raft_consensus.cc:2258] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.543618 32477 tablet_replica.cc:330] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:27.543917 32459 ts_tablet_manager.cc:1875] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.543994 32477 raft_consensus.cc:2229] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:27.544364 32477 raft_consensus.cc:2258] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.545744 32477 ts_tablet_manager.cc:1875] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.546228 32459 ts_tablet_manager.cc:1888] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:27.546300 32459 log.cc:1196] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/ff33763a720b48edbbf5ddc96861a615
I20240502 14:00:27.546573 32459 ts_tablet_manager.cc:1909] T ff33763a720b48edbbf5ddc96861a615 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:27.548749 32477 ts_tablet_manager.cc:1888] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:27.548836 32477 log.cc:1196] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/ff33763a720b48edbbf5ddc96861a615
I20240502 14:00:27.549132 32477 ts_tablet_manager.cc:1909] T ff33763a720b48edbbf5ddc96861a615 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:27.550000 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet ff33763a720b48edbbf5ddc96861a615 (table rebalancer_dynamic_table_new_name [id=0a940a9153844ba8ad7e8f05d3ff41b9]) successfully deleted
I20240502 14:00:27.550253 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet ff33763a720b48edbbf5ddc96861a615 (table rebalancer_dynamic_table_new_name [id=0a940a9153844ba8ad7e8f05d3ff41b9]) successfully deleted
I20240502 14:00:27.550287 32478 tablet_replica.cc:330] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:27.550565 32478 raft_consensus.cc:2229] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.550767 32478 raft_consensus.cc:2258] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.552008 32478 ts_tablet_manager.cc:1875] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.554428 32478 ts_tablet_manager.cc:1888] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:27.554507 32478 log.cc:1196] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/ff33763a720b48edbbf5ddc96861a615
I20240502 14:00:27.554908 32478 ts_tablet_manager.cc:1909] T ff33763a720b48edbbf5ddc96861a615 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:27.556066 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet ff33763a720b48edbbf5ddc96861a615 (table rebalancer_dynamic_table_new_name [id=0a940a9153844ba8ad7e8f05d3ff41b9]) successfully deleted
I20240502 14:00:27.608167 32323 raft_consensus.cc:482] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:27.608310 32323 raft_consensus.cc:504] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.609632 32323 leader_election.cc:288] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:27.615284 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e9d8ef4f754df3817cc4a981549e02" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:27.615348 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e9d8ef4f754df3817cc4a981549e02" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:27.615617 32255 raft_consensus.cc:2454] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:27.615638 31992 raft_consensus.cc:2454] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:27.616051 31796 leader_election.cc:302] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:27.616322 32323 raft_consensus.cc:2781] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:27.616390 32323 raft_consensus.cc:482] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:27.616430 32323 raft_consensus.cc:3037] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.617468 32323 raft_consensus.cc:504] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.617835 32323 leader_election.cc:288] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:27.619923 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e9d8ef4f754df3817cc4a981549e02" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:27.620064 31992 raft_consensus.cc:3037] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.620318 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e9d8ef4f754df3817cc4a981549e02" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:27.620437 32255 raft_consensus.cc:3037] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.621189 31992 raft_consensus.cc:2454] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:27.621469 31794 leader_election.cc:302] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:27.621558 32255 raft_consensus.cc:2454] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:27.621649 32323 raft_consensus.cc:2781] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:27.621842 32323 raft_consensus.cc:686] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:27.622079 32323 consensus_queue.cc:237] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.624068 29325 catalog_manager.cc:5484] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:27.629778 32469 raft_consensus.cc:482] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:27.629904 32469 raft_consensus.cc:504] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.630334 32469 leader_election.cc:288] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
W20240502 14:00:27.630875 32302 tablet.cc:2369] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:27.638593 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a84488994c47ba9436ea7161d9b2e3" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:27.638813 32119 raft_consensus.cc:2454] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:27.639413 29797 leader_election.cc:302] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:27.639586 32469 raft_consensus.cc:2781] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:27.639637 32469 raft_consensus.cc:482] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:27.639604 32470 raft_consensus.cc:482] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:27.639674 32469 raft_consensus.cc:3037] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.639699 32470 raft_consensus.cc:504] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.640681 32469 raft_consensus.cc:504] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.640888 32470 leader_election.cc:288] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:27.641007 32469 leader_election.cc:288] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:27.643036 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a84488994c47ba9436ea7161d9b2e3" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:27.643154 32119 raft_consensus.cc:3037] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.643918 29320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:27.644212 32119 raft_consensus.cc:2454] T e4a84488994c47ba9436ea7161d9b2e3 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:27.644708 29797 leader_election.cc:302] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:27.645263 32469 raft_consensus.cc:2781] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:27.645342 32469 raft_consensus.cc:686] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:27.645470 32469 consensus_queue.cc:237] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.650151 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a84488994c47ba9436ea7161d9b2e3" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:27.650327 31992 raft_consensus.cc:2454] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:27.650192 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a84488994c47ba9436ea7161d9b2e3" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:27.650517 31991 raft_consensus.cc:3037] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.651604 31991 raft_consensus.cc:2454] T e4a84488994c47ba9436ea7161d9b2e3 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:27.652940 29324 catalog_manager.cc:5484] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:27.654098 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37]), partition=
I20240502 14:00:27.654109 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37]), partition=
I20240502 14:00:27.654361 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.654369 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.656461 32462 tablet_bootstrap.cc:492] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:27.656935 32451 tablet_bootstrap.cc:492] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:27.657419 32462 tablet_bootstrap.cc:654] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.657720 32451 tablet_bootstrap.cc:654] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.658066 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d48f827a6243448d344ee44a54482d" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:27.658211 29470 raft_consensus.cc:2454] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:27.658443 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37]), partition=
I20240502 14:00:27.658669 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.658824 32190 leader_election.cc:302] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:27.660367 32451 tablet_bootstrap.cc:492] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:27.660455 32451 ts_tablet_manager.cc:1367] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:27.660815 32470 raft_consensus.cc:2781] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:27.660872 32470 raft_consensus.cc:482] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:27.660828 32451 raft_consensus.cc:348] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.660913 32470 raft_consensus.cc:3037] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.660930 32451 raft_consensus.cc:374] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.660964 32451 raft_consensus.cc:729] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.661082 32451 consensus_queue.cc:260] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.661376 32451 ts_tablet_manager.cc:1398] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.661760 32470 raft_consensus.cc:504] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.662114 32470 leader_election.cc:288] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:27.662428 32463 tablet_bootstrap.cc:492] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:27.662516 32462 tablet_bootstrap.cc:492] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:27.662577 32462 ts_tablet_manager.cc:1367] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:27.663290 32463 tablet_bootstrap.cc:654] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.663903 32462 raft_consensus.cc:348] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.664005 32462 raft_consensus.cc:374] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.664036 32462 raft_consensus.cc:729] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.664158 32462 consensus_queue.cc:260] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.664403 32462 ts_tablet_manager.cc:1398] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:27.665088 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d48f827a6243448d344ee44a54482d" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:27.665202 29470 raft_consensus.cc:3037] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.666152 29470 raft_consensus.cc:2454] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:27.666533 32190 leader_election.cc:302] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:27.667290 32470 raft_consensus.cc:2781] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:27.667536 32470 raft_consensus.cc:686] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:27.667882 32470 consensus_queue.cc:237] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.670941 29325 catalog_manager.cc:5484] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:27.673741 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d48f827a6243448d344ee44a54482d" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:27.673909 31861 raft_consensus.cc:2454] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:27.674065 32463 tablet_bootstrap.cc:492] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:27.674166 32463 ts_tablet_manager.cc:1367] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:27.674163 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d48f827a6243448d344ee44a54482d" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:27.674266 31860 raft_consensus.cc:3037] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.674664 32463 raft_consensus.cc:348] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.674789 32463 raft_consensus.cc:374] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.674825 32463 raft_consensus.cc:729] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.674974 32463 consensus_queue.cc:260] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.675230 31860 raft_consensus.cc:2454] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:27.675242 32463 ts_tablet_manager.cc:1398] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.698210 31972 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.707926 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:27.709587 29450 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:27.735725 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:27.740283 32103 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:27.740971 29580 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:27.741231 31841 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:27.744846 29844 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:27.765455 32467 raft_consensus.cc:482] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:27.765592 32467 raft_consensus.cc:504] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.766677 32467 leader_election.cc:288] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:27.771893 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4af51bd674f80b2d02dbaf4ccc6f2" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:27.772056 29470 raft_consensus.cc:2454] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:27.772434 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4af51bd674f80b2d02dbaf4ccc6f2" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:27.772588 32255 raft_consensus.cc:2454] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:27.772842 31927 leader_election.cc:302] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:27.773159 32467 raft_consensus.cc:2781] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:27.773218 32467 raft_consensus.cc:482] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:27.773257 32467 raft_consensus.cc:3037] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.774240 32467 raft_consensus.cc:504] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.774581 32467 leader_election.cc:288] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:27.775111 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4af51bd674f80b2d02dbaf4ccc6f2" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:27.775224 29470 raft_consensus.cc:3037] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.776212 29470 raft_consensus.cc:2454] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:27.776713 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4af51bd674f80b2d02dbaf4ccc6f2" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:27.776824 32255 raft_consensus.cc:3037] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.777771 32255 raft_consensus.cc:2454] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:27.778183 31927 leader_election.cc:302] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:27.778487 32467 raft_consensus.cc:2781] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:27.778703 32467 raft_consensus.cc:686] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:27.779050 32467 consensus_queue.cc:237] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.782111 29325 catalog_manager.cc:5484] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:27.830142 29600 consensus_queue.cc:237] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:27.831791 29732 raft_consensus.cc:1261] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.832108 29863 raft_consensus.cc:1261] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.832379 31131 consensus_queue.cc:1035] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.832573 31659 consensus_queue.cc:1035] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.834769 31131 raft_consensus.cc:2932] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:27.836930 29732 raft_consensus.cc:2932] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:27.836910 29863 raft_consensus.cc:2932] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
W20240502 14:00:27.837944 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0867c6a22c2c4efd9a9babff371cc8ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:27.838110 29325 catalog_manager.cc:5484] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:27.859280 29863 consensus_queue.cc:237] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:27.861013 29600 raft_consensus.cc:1261] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.861435 29470 raft_consensus.cc:1261] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.861887 31463 consensus_queue.cc:1035] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.862236 32469 consensus_queue.cc:1035] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.864895 32469 raft_consensus.cc:2932] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:27.865342 29601 raft_consensus.cc:2932] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
W20240502 14:00:27.867100 29799 consensus_peers.cc:485] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3ae9aa7809c4372aa9d322458baa6b8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:27.867585 29470 raft_consensus.cc:2932] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:27.867831 29326 catalog_manager.cc:5484] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:27.889760 29600 consensus_queue.cc:237] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:27.891283 29863 raft_consensus.cc:1261] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.891391 29470 raft_consensus.cc:1261] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.891723 32561 consensus_queue.cc:1035] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.892309 32561 consensus_queue.cc:1035] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.895282 31131 raft_consensus.cc:2932] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
W20240502 14:00:27.895536 29535 consensus_peers.cc:485] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65cceaa60c7e470ba8b8d205d026625b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:27.896749 29863 raft_consensus.cc:2932] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:27.898182 29322 catalog_manager.cc:5484] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:27.899888 29470 raft_consensus.cc:2932] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:27.907760 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:27.913214 29470 raft_consensus.cc:1261] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:27.913269 32255 raft_consensus.cc:1261] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:27.913797 32467 consensus_queue.cc:1035] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:27.914077 32542 consensus_queue.cc:1035] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:27.917302 29325 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (table rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37]). Expected version 1 got 0
I20240502 14:00:27.917354 32623 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:27.918107 29322 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (table rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37]). Expected version 1 got 0
I20240502 14:00:27.918397 32623 tablet.cc:1620] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:27.918314 32624 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:27.919725 32624 tablet.cc:1620] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:27.922672 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37] alter complete (version 1)
I20240502 14:00:27.924120 29732 consensus_queue.cc:237] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:27.925640 29862 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.926328 29600 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.926752 31764 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.928170 31764 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.930346 31764 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:27.930750 29600 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:27.930997 32630 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:27.931766 29326 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (table rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37]). Expected version 1 got 0
W20240502 14:00:27.932343 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 463dddd3473a42a68b488e1c465e997f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:27.931951 32630 tablet.cc:1620] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:27.932533 32620 mvcc.cc:205] Tried to move back new op lower bound from 7023240920726695936 to 7023240920185655296. Current Snapshot: MvccSnapshot[applied={T|T < 7023240920726695936}]
I20240502 14:00:27.933676 29326 catalog_manager.cc:5484] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:27.934780 29862 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:27.946067 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 14:00:27.946229 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 14:00:27.948908 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet 65cceaa60c7e470ba8b8d205d026625b
I20240502 14:00:27.949349 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet bd9618b854814e11a3c6ae7504821d35
I20240502 14:00:27.949386 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cceaa60c7e470ba8b8d205d026625b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:27.949568 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cceaa60c7e470ba8b8d205d026625b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:27.949590 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 58a2db3a7fca4420a94891cd96c54ec3
I20240502 14:00:27.949637 32458 tablet_replica.cc:330] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:27.949725 32460 tablet_replica.cc:330] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:27.949771 32458 raft_consensus.cc:2229] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:27.949832 32460 raft_consensus.cc:2229] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.949973 32460 raft_consensus.cc:2258] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.950057 32458 raft_consensus.cc:2258] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.950268 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cceaa60c7e470ba8b8d205d026625b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:27.950371 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9618b854814e11a3c6ae7504821d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:27.950431 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a2db3a7fca4420a94891cd96c54ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:27.950572 32457 tablet_replica.cc:330] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:27.950680 32457 raft_consensus.cc:2229] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.950676 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9618b854814e11a3c6ae7504821d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:27.950875 32457 raft_consensus.cc:2258] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.950834 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cceaa60c7e470ba8b8d205d026625b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:27.951013 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9618b854814e11a3c6ae7504821d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:27.951049 32459 tablet_replica.cc:330] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:27.951102 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a2db3a7fca4420a94891cd96c54ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:27.951208 32459 raft_consensus.cc:2229] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.951342 32459 raft_consensus.cc:2258] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.951409 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a2db3a7fca4420a94891cd96c54ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:27 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:27.951655 32460 ts_tablet_manager.cc:1875] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.951866 32459 ts_tablet_manager.cc:1875] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 14:00:27.951934 29313 catalog_manager.cc:4797] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): delete failed for tablet 65cceaa60c7e470ba8b8d205d026625b because the tablet was not found. No further retry: Not found: Tablet not found: 65cceaa60c7e470ba8b8d205d026625b
I20240502 14:00:27.953377 32460 ts_tablet_manager.cc:1888] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:27.953431 32460 log.cc:1196] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/65cceaa60c7e470ba8b8d205d026625b
I20240502 14:00:27.953637 32460 ts_tablet_manager.cc:1909] T 65cceaa60c7e470ba8b8d205d026625b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:27.953899 32459 ts_tablet_manager.cc:1888] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.953955 32459 log.cc:1196] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd9618b854814e11a3c6ae7504821d35
I20240502 14:00:27.954192 32459 ts_tablet_manager.cc:1909] T bd9618b854814e11a3c6ae7504821d35 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:27.954223 32458 ts_tablet_manager.cc:1875] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.954304 32460 tablet_replica.cc:330] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:27.954435 32460 raft_consensus.cc:2229] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.954555 32460 raft_consensus.cc:2258] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.955327 32457 ts_tablet_manager.cc:1875] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.955428 32460 ts_tablet_manager.cc:1875] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.956349 32458 ts_tablet_manager.cc:1888] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:27.956413 32458 log.cc:1196] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/65cceaa60c7e470ba8b8d205d026625b
I20240502 14:00:27.956663 32458 ts_tablet_manager.cc:1909] T 65cceaa60c7e470ba8b8d205d026625b P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:27.957015 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 65cceaa60c7e470ba8b8d205d026625b (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.957295 32460 ts_tablet_manager.cc:1888] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.957304 32457 ts_tablet_manager.cc:1888] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:27.957352 32460 log.cc:1196] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/bd9618b854814e11a3c6ae7504821d35
I20240502 14:00:27.957367 32457 log.cc:1196] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/65cceaa60c7e470ba8b8d205d026625b
I20240502 14:00:27.957626 32457 ts_tablet_manager.cc:1909] T 65cceaa60c7e470ba8b8d205d026625b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:27.957621 32460 ts_tablet_manager.cc:1909] T bd9618b854814e11a3c6ae7504821d35 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:27.957784 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet bd9618b854814e11a3c6ae7504821d35 (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.958369 32460 tablet_replica.cc:330] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:27.958446 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet bd9618b854814e11a3c6ae7504821d35 (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.958469 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 65cceaa60c7e470ba8b8d205d026625b (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.958635 32460 raft_consensus.cc:2229] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.958662 32457 tablet_replica.cc:330] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:27.958734 32458 tablet_replica.cc:330] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:27.958796 32457 raft_consensus.cc:2229] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:27.958801 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 65cceaa60c7e470ba8b8d205d026625b (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.958879 32460 raft_consensus.cc:2258] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.958920 32458 raft_consensus.cc:2229] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:27.959086 32457 raft_consensus.cc:2258] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.959153 32458 raft_consensus.cc:2258] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.959414 32460 ts_tablet_manager.cc:1875] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.959589 32457 ts_tablet_manager.cc:1875] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.959733 32458 ts_tablet_manager.cc:1875] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.961299 32460 ts_tablet_manager.cc:1888] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.961426 32460 log.cc:1196] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/58a2db3a7fca4420a94891cd96c54ec3
I20240502 14:00:27.961450 32457 ts_tablet_manager.cc:1888] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.961541 32457 log.cc:1196] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd9618b854814e11a3c6ae7504821d35
I20240502 14:00:27.961740 32460 ts_tablet_manager.cc:1909] T 58a2db3a7fca4420a94891cd96c54ec3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:27.961856 32457 ts_tablet_manager.cc:1909] T bd9618b854814e11a3c6ae7504821d35 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:27.962368 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 58a2db3a7fca4420a94891cd96c54ec3 (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.962656 32458 ts_tablet_manager.cc:1888] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.962720 32458 log.cc:1196] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/58a2db3a7fca4420a94891cd96c54ec3
I20240502 14:00:27.962965 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet bd9618b854814e11a3c6ae7504821d35 (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.963024 32458 ts_tablet_manager.cc:1909] T 58a2db3a7fca4420a94891cd96c54ec3 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:27.962888 32457 tablet_replica.cc:330] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:27.963188 32457 raft_consensus.cc:2229] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:27.963301 32457 raft_consensus.cc:2258] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:27.963624 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 58a2db3a7fca4420a94891cd96c54ec3 (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.963719 32457 ts_tablet_manager.cc:1875] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:27.965385 32457 ts_tablet_manager.cc:1888] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:27.965456 32457 log.cc:1196] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/58a2db3a7fca4420a94891cd96c54ec3
I20240502 14:00:27.965694 32457 ts_tablet_manager.cc:1909] T 58a2db3a7fca4420a94891cd96c54ec3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:27.967262 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 58a2db3a7fca4420a94891cd96c54ec3 (table rebalance_test_table_2 [id=96a5391ddb3248208c3a309b102ba4ee]) successfully deleted
I20240502 14:00:27.978731 29862 consensus_queue.cc:237] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:27.980099 29732 raft_consensus.cc:1261] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:27.980185 29470 raft_consensus.cc:1261] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:27.980329 29797 consensus_peers.cc:485] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad03c8bc0c994bc6a9e17c1206f0d8b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:27.980576 32469 consensus_queue.cc:1035] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.980741 31463 consensus_queue.cc:1035] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:27.982620 31463 raft_consensus.cc:2932] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:27.983047 29732 raft_consensus.cc:2932] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:27.985431 29470 raft_consensus.cc:2932] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:27.985855 29322 catalog_manager.cc:5484] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.005599 29862 consensus_queue.cc:237] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.006918 29732 raft_consensus.cc:1261] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.006989 29470 raft_consensus.cc:1261] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:28.007315 29799 consensus_peers.cc:485] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e227b307a0f44a088bf50899d59f2118. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.007346 31463 consensus_queue.cc:1035] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.007895 31463 consensus_queue.cc:1035] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.008813 32517 consensus_queue.cc:1035] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.011420 31463 raft_consensus.cc:2932] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.012905 32323 consensus_queue.cc:1035] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.013659 29470 raft_consensus.cc:2932] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.013720 29732 raft_consensus.cc:2932] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.016865 29325 catalog_manager.cc:5484] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.027372 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:28.033160 32623 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:28.033485 32623 tablet.cc:1620] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:28.034969 32624 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:28.035272 32624 tablet.cc:1620] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:28.035766 32630 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:28.036065 32630 tablet.cc:1620] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:28.039140 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=6af7cc0fdcad4feb9f355146a9bdbb37] alter complete (version 2)
I20240502 14:00:28.041671 29862 consensus_queue.cc:237] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.043371 29470 raft_consensus.cc:1261] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.043416 29600 raft_consensus.cc:1261] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.043862 32672 consensus_queue.cc:1035] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.044025 32469 consensus_queue.cc:1035] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:28.044461 29797 consensus_peers.cc:485] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36daa6b40d8a48b8956b67a97833eff5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.048053 32469 raft_consensus.cc:2932] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.048499 29601 raft_consensus.cc:2932] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.050055 29470 raft_consensus.cc:2932] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.052192 29326 catalog_manager.cc:5484] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.071878 29732 consensus_queue.cc:237] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.073410 29862 raft_consensus.cc:1261] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.073459 29601 raft_consensus.cc:1261] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:28.073694 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 209e6922777b49ddb87a3dadf65d651a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.073817 32631 consensus_queue.cc:1035] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.074018 32632 consensus_queue.cc:1035] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.075704 32631 raft_consensus.cc:2932] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.076126 29601 raft_consensus.cc:2932] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.078055 29862 raft_consensus.cc:2932] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.078774 29322 catalog_manager.cc:5484] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.096642 29601 consensus_queue.cc:237] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.097924 29470 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:28.098237 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253ea421888249d180928ba0aef28ddd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.098397 29733 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.098696 31659 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.099011 32602 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.101135 32602 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.101521 29470 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.102869 29733 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.104332 29325 catalog_manager.cc:5484] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.124066 29862 consensus_queue.cc:237] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.125098 32581 consensus_queue.cc:1035] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.125808 29601 raft_consensus.cc:1261] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.126283 32672 consensus_queue.cc:1035] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:28.127952 29799 consensus_peers.cc:485] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4697431776e44a1907cf4b0865414da. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.128225 29470 raft_consensus.cc:1261] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.128715 32672 consensus_queue.cc:1035] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.130648 32581 consensus_queue.cc:1035] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.130786 32469 raft_consensus.cc:2932] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.131474 29470 raft_consensus.cc:2932] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.133388 29601 raft_consensus.cc:2932] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.134958 29326 catalog_manager.cc:5484] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.147316 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:28.152742 32623 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:28.153160 32624 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:28.153499 32630 tablet.cc:1722] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:28.159478 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=6af7cc0fdcad4feb9f355146a9bdbb37] alter complete (version 3)
I20240502 14:00:28.164091 29470 consensus_queue.cc:237] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.165710 29862 raft_consensus.cc:1261] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.165843 29601 raft_consensus.cc:1261] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.166157 31578 consensus_queue.cc:1035] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.166790 31578 consensus_queue.cc:1035] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:28.168928 29404 consensus_peers.cc:485] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6767168f6f2b4e15aab7e64b0eab57c3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.169243 301 raft_consensus.cc:2932] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.169669 29601 raft_consensus.cc:2932] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.171226 29862 raft_consensus.cc:2932] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.172639 29324 catalog_manager.cc:5484] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.182566 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:28.189081 31841 tablet_service.cc:1467] Processing CreateTablet for tablet e574c148957f44bf9c2935e955de99aa (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.189081 31972 tablet_service.cc:1467] Processing CreateTablet for tablet e574c148957f44bf9c2935e955de99aa (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.189389 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e574c148957f44bf9c2935e955de99aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.189401 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e574c148957f44bf9c2935e955de99aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.191659 29580 tablet_service.cc:1467] Processing CreateTablet for tablet e9f48a525d8c4a0783f0450aa6ec7cb0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.191967 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f48a525d8c4a0783f0450aa6ec7cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.191926 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 3a34a1381cf84af6b560101d95447e6d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.192179 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a34a1381cf84af6b560101d95447e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.192284 32235 tablet_service.cc:1467] Processing CreateTablet for tablet e574c148957f44bf9c2935e955de99aa (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.192284 32234 tablet_service.cc:1467] Processing CreateTablet for tablet e9f48a525d8c4a0783f0450aa6ec7cb0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.192600 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e574c148957f44bf9c2935e955de99aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.192718 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f48a525d8c4a0783f0450aa6ec7cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.194552 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 3a34a1381cf84af6b560101d95447e6d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.194784 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a34a1381cf84af6b560101d95447e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.195029 317 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:28.195986 317 tablet_bootstrap.cc:654] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.196213 318 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:28.197032 317 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:28.197116 318 tablet_bootstrap.cc:654] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.197239 317 ts_tablet_manager.cc:1367] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:28.197798 317 raft_consensus.cc:348] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.198015 317 raft_consensus.cc:374] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.198074 31972 tablet_service.cc:1467] Processing CreateTablet for tablet e9f48a525d8c4a0783f0450aa6ec7cb0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.198079 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 3a34a1381cf84af6b560101d95447e6d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.198392 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f48a525d8c4a0783f0450aa6ec7cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.198894 319 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:28.199232 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a34a1381cf84af6b560101d95447e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.199996 319 tablet_bootstrap.cc:654] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.198103 317 raft_consensus.cc:729] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.200484 317 consensus_queue.cc:260] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.200776 317 ts_tablet_manager.cc:1398] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:28.201002 317 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:28.201536 319 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:28.201632 319 ts_tablet_manager.cc:1367] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:28.201910 317 tablet_bootstrap.cc:654] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.202064 319 raft_consensus.cc:348] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.202181 319 raft_consensus.cc:374] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.202522 319 raft_consensus.cc:729] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.202551 320 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.196594 316 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:28.202754 319 consensus_queue.cc:260] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.203166 319 ts_tablet_manager.cc:1398] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.203897 318 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:28.203958 317 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:28.203994 318 ts_tablet_manager.cc:1367] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:00:28.204056 317 ts_tablet_manager.cc:1367] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:28.204365 318 raft_consensus.cc:348] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.204485 318 raft_consensus.cc:374] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.204546 318 raft_consensus.cc:729] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.204522 317 raft_consensus.cc:348] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.204627 317 raft_consensus.cc:374] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.204658 317 raft_consensus.cc:729] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.205097 318 consensus_queue.cc:260] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.205564 320 tablet_bootstrap.cc:654] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.205878 317 consensus_queue.cc:260] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.206753 317 ts_tablet_manager.cc:1398] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:28.207010 317 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:28.207513 316 tablet_bootstrap.cc:654] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.207610 32470 consensus_queue.cc:1035] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.207988 317 tablet_bootstrap.cc:654] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.209054 320 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:28.209409 320 ts_tablet_manager.cc:1367] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:28.209566 318 ts_tablet_manager.cc:1398] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:28.210718 320 raft_consensus.cc:348] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.210912 320 raft_consensus.cc:374] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.210999 320 raft_consensus.cc:729] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.211169 320 consensus_queue.cc:260] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.211453 320 ts_tablet_manager.cc:1398] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:28.211618 320 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.213404 320 tablet_bootstrap.cc:654] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.213639 29733 consensus_queue.cc:237] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.214552 317 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:28.214709 317 ts_tablet_manager.cc:1367] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:28.215598 317 raft_consensus.cc:348] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.215647 29862 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.215778 317 raft_consensus.cc:374] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.215992 317 raft_consensus.cc:729] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.216159 317 consensus_queue.cc:260] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.216432 317 ts_tablet_manager.cc:1398] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:28.216483 32470 consensus_queue.cc:1035] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.217408 320 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:28.217552 320 ts_tablet_manager.cc:1367] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:28.218670 320 raft_consensus.cc:348] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.218757 32631 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.218837 320 raft_consensus.cc:374] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.218999 320 raft_consensus.cc:729] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.219169 320 consensus_queue.cc:260] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.219910 320 ts_tablet_manager.cc:1398] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:28.220089 320 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.220911 29601 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.221230 320 tablet_bootstrap.cc:654] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.222613 32631 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.222672 316 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:28.222774 316 ts_tablet_manager.cc:1367] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240502 14:00:28.223289 316 raft_consensus.cc:348] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.223418 316 raft_consensus.cc:374] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.223491 316 raft_consensus.cc:729] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.223706 316 consensus_queue.cc:260] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.224566 316 ts_tablet_manager.cc:1398] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 14:00:28.226145 29666 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bdf925a9c4d456fb3b4ef448a2154c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.226903 320 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:28.227082 320 ts_tablet_manager.cc:1367] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:28.227293 32626 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.227738 29601 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.227838 320 raft_consensus.cc:348] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.228122 320 raft_consensus.cc:374] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.228242 320 raft_consensus.cc:729] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.228407 320 consensus_queue.cc:260] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.228771 320 ts_tablet_manager.cc:1398] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:28.230146 29862 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.231335 29324 catalog_manager.cc:5484] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.253355 29601 consensus_queue.cc:237] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.254735 29470 raft_consensus.cc:1261] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.254878 29733 raft_consensus.cc:1261] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.255252 31131 consensus_queue.cc:1035] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.255477 32561 consensus_queue.cc:1035] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.257295 31131 raft_consensus.cc:2932] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.257743 29733 raft_consensus.cc:2932] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.259130 29470 raft_consensus.cc:2932] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.260222 29328 catalog_manager.cc:5484] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
W20240502 14:00:28.261574 29537 consensus_peers.cc:485] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba65f51fa3f64a479e1e62042c1ca64e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.262490 29324 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:28.262619 29324 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:28.265591 29324 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 10c4af51bd674f80b2d02dbaf4ccc6f2
I20240502 14:00:28.266132 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:28.266294 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:28.266396 32643 tablet_replica.cc:330] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:28.266722 32643 raft_consensus.cc:2229] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:28.266832 363 tablet_replica.cc:330] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:28.267022 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:28.267122 32643 raft_consensus.cc:2258] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.267156 363 raft_consensus.cc:2229] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.267216 32457 tablet_replica.cc:330] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:28.267320 363 raft_consensus.cc:2258] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.267369 32457 raft_consensus.cc:2229] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.267514 32457 raft_consensus.cc:2258] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.268121 32457 ts_tablet_manager.cc:1875] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.268235 363 ts_tablet_manager.cc:1875] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.268235 32643 ts_tablet_manager.cc:1875] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.270001 32457 ts_tablet_manager.cc:1888] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.270071 32457 log.cc:1196] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/10c4af51bd674f80b2d02dbaf4ccc6f2
I20240502 14:00:28.270341 32457 ts_tablet_manager.cc:1909] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:28.270977 32643 ts_tablet_manager.cc:1888] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.271044 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (table rebalancer_dynamic_table_new_name [id=6af7cc0fdcad4feb9f355146a9bdbb37]) successfully deleted
I20240502 14:00:28.271200 32643 log.cc:1196] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/10c4af51bd674f80b2d02dbaf4ccc6f2
I20240502 14:00:28.271672 32643 ts_tablet_manager.cc:1909] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:28.272835 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (table rebalancer_dynamic_table_new_name [id=6af7cc0fdcad4feb9f355146a9bdbb37]) successfully deleted
I20240502 14:00:28.272882 363 ts_tablet_manager.cc:1888] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.273021 363 log.cc:1196] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/10c4af51bd674f80b2d02dbaf4ccc6f2
I20240502 14:00:28.273380 363 ts_tablet_manager.cc:1909] T 10c4af51bd674f80b2d02dbaf4ccc6f2 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:28.274344 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 10c4af51bd674f80b2d02dbaf4ccc6f2 (table rebalancer_dynamic_table_new_name [id=6af7cc0fdcad4feb9f355146a9bdbb37]) successfully deleted
I20240502 14:00:28.285595 29601 consensus_queue.cc:237] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.287030 29471 raft_consensus.cc:1261] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.287027 29862 raft_consensus.cc:1261] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.287483 31131 consensus_queue.cc:1035] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.287660 32561 consensus_queue.cc:1035] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:28.287914 29535 consensus_peers.cc:485] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3bf63852df44cf1b762b7a981428c97. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.289429 31131 raft_consensus.cc:2932] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.289837 29862 raft_consensus.cc:2932] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.292572 29324 catalog_manager.cc:5484] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.295485 29471 raft_consensus.cc:2932] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.317591 31659 raft_consensus.cc:482] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.317692 31659 raft_consensus.cc:504] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.318010 31659 leader_election.cc:288] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:28.318686 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f48a525d8c4a0783f0450aa6ec7cb0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:28.318823 32255 raft_consensus.cc:2454] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:28.319003 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f48a525d8c4a0783f0450aa6ec7cb0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:28.319169 31991 raft_consensus.cc:2454] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:28.319329 29537 leader_election.cc:302] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:28.319851 32561 raft_consensus.cc:2781] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.319923 32561 raft_consensus.cc:482] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.319962 32561 raft_consensus.cc:3037] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.321096 32561 raft_consensus.cc:504] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.321486 32561 leader_election.cc:288] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:28.322072 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f48a525d8c4a0783f0450aa6ec7cb0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:28.322197 31991 raft_consensus.cc:3037] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.322494 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f48a525d8c4a0783f0450aa6ec7cb0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:28.322603 32255 raft_consensus.cc:3037] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.323215 31991 raft_consensus.cc:2454] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:28.323657 32255 raft_consensus.cc:2454] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:28.324195 29537 leader_election.cc:302] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:28.324568 32561 raft_consensus.cc:2781] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.324649 32561 raft_consensus.cc:686] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:28.324772 32561 consensus_queue.cc:237] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.326972 29329 catalog_manager.cc:5484] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.333356 29471 consensus_queue.cc:237] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.334780 29601 raft_consensus.cc:1261] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.334808 29862 raft_consensus.cc:1261] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:28.334968 29404 consensus_peers.cc:485] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59235c5f3daf4ff1afeab83efe018405. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.335297 31578 consensus_queue.cc:1035] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.335484 32173 consensus_queue.cc:1035] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.337225 31578 raft_consensus.cc:2932] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.337667 29601 raft_consensus.cc:2932] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.340340 29329 catalog_manager.cc:5484] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.340379 29862 raft_consensus.cc:2932] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.362577 29862 consensus_queue.cc:237] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.364099 29601 raft_consensus.cc:1261] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:28.364199 29797 consensus_peers.cc:485] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3517a9ff83644ccb5cd4c0ebbd6e929. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.364478 29733 raft_consensus.cc:1261] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.364568 32581 consensus_queue.cc:1035] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.364903 31463 consensus_queue.cc:1035] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.366644 32581 raft_consensus.cc:2932] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.367069 29733 raft_consensus.cc:2932] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.368901 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:28.370090 29601 raft_consensus.cc:2932] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.370615 29324 catalog_manager.cc:5484] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.372557 32749 raft_consensus.cc:482] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.372687 32749 raft_consensus.cc:504] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.373210 32749 leader_election.cc:288] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:28.377914 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a34a1381cf84af6b560101d95447e6d" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:28.378055 32255 raft_consensus.cc:2454] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:28.378333 32058 leader_election.cc:302] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:28.378582 32749 raft_consensus.cc:2781] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.378633 32749 raft_consensus.cc:482] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.378680 32749 raft_consensus.cc:3037] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.379796 32749 raft_consensus.cc:504] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.380020 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a34a1381cf84af6b560101d95447e6d" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:28.380170 31991 raft_consensus.cc:2454] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:28.380182 32749 leader_election.cc:288] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:28.380692 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a34a1381cf84af6b560101d95447e6d" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:28.380784 31991 raft_consensus.cc:3037] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.381471 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 1cff896decf14fca8307ff326a8f64ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457]), partition=
I20240502 14:00:28.381568 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 1cff896decf14fca8307ff326a8f64ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457]), partition=
I20240502 14:00:28.381687 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cff896decf14fca8307ff326a8f64ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.381762 31991 raft_consensus.cc:2454] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:28.381829 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cff896decf14fca8307ff326a8f64ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.382048 32056 leader_election.cc:302] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:28.382249 32749 raft_consensus.cc:2781] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.382347 32749 raft_consensus.cc:686] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:28.382481 32749 consensus_queue.cc:237] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.383839 424 tablet_bootstrap.cc:492] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:28.383996 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 1cff896decf14fca8307ff326a8f64ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457]), partition=
I20240502 14:00:28.384259 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cff896decf14fca8307ff326a8f64ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.384514 318 tablet_bootstrap.cc:492] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:28.384706 424 tablet_bootstrap.cc:654] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.385449 318 tablet_bootstrap.cc:654] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.385915 424 tablet_bootstrap.cc:492] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:28.386000 424 ts_tablet_manager.cc:1367] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:28.386296 320 tablet_bootstrap.cc:492] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.386831 424 raft_consensus.cc:348] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.386982 424 raft_consensus.cc:374] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.387022 424 raft_consensus.cc:729] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.387156 424 consensus_queue.cc:260] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.387238 320 tablet_bootstrap.cc:654] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.387468 424 ts_tablet_manager.cc:1398] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:28.388075 29329 catalog_manager.cc:5484] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:28.388293 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a34a1381cf84af6b560101d95447e6d" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:28.388403 32255 raft_consensus.cc:3037] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.389030 318 tablet_bootstrap.cc:492] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:28.389123 318 ts_tablet_manager.cc:1367] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:28.389312 32255 raft_consensus.cc:2454] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:28.389596 318 raft_consensus.cc:348] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.389683 318 raft_consensus.cc:374] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.389703 318 raft_consensus.cc:729] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.389789 318 consensus_queue.cc:260] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.390097 318 ts_tablet_manager.cc:1398] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.392848 320 tablet_bootstrap.cc:492] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:28.392951 320 ts_tablet_manager.cc:1367] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:28.393437 320 raft_consensus.cc:348] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.393527 320 raft_consensus.cc:374] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.393558 320 raft_consensus.cc:729] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.393671 320 consensus_queue.cc:260] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.393903 320 ts_tablet_manager.cc:1398] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.410800 29471 consensus_queue.cc:237] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.412691 29601 raft_consensus.cc:1261] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.413177 32173 consensus_queue.cc:1035] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.413931 29733 raft_consensus.cc:1261] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.414902 301 consensus_queue.cc:1035] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:28.415781 29406 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2c754e60c1d4f6f95a807e9644bd21f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.416657 301 raft_consensus.cc:2932] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.417075 29733 raft_consensus.cc:2932] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.418454 29601 raft_consensus.cc:2932] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.419934 441 ts_tablet_manager.cc:902] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:28.420742 441 tablet_copy_client.cc:308] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:28.420974 29324 catalog_manager.cc:5484] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.425513 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 463dddd3473a42a68b488e1c465e997f from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:57583)
I20240502 14:00:28.425623 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 463dddd3473a42a68b488e1c465e997f from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:57583: session id = f0e01c4f581843e9ad4ced97f0072dc4-463dddd3473a42a68b488e1c465e997f
I20240502 14:00:28.426914 29753 tablet_copy_source_session.cc:205] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.428541 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dddd3473a42a68b488e1c465e997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.430811 441 tablet_copy_client.cc:783] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.431005 441 tablet_copy_client.cc:647] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.432669 441 tablet_copy_client.cc:523] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.434119 441 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:28.437361 457 ts_tablet_manager.cc:902] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:28.438206 457 tablet_copy_client.cc:308] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:28.443766 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 0867c6a22c2c4efd9a9babff371cc8ba from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37007)
I20240502 14:00:28.443899 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 0867c6a22c2c4efd9a9babff371cc8ba from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37007: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-0867c6a22c2c4efd9a9babff371cc8ba
I20240502 14:00:28.445242 29622 tablet_copy_source_session.cc:205] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.446707 29862 consensus_queue.cc:237] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.446956 457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0867c6a22c2c4efd9a9babff371cc8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.448534 29733 raft_consensus.cc:1261] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.448577 29471 raft_consensus.cc:1261] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:28.448904 29797 consensus_peers.cc:485] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ebf61381a92430caa0c38cc04d14cb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.449105 32581 consensus_queue.cc:1035] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.449278 32469 consensus_queue.cc:1035] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.450234 457 tablet_copy_client.cc:783] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.450493 457 tablet_copy_client.cc:647] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.451588 29329 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 14:00:28.451725 29329 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 14:00:28.452379 457 tablet_copy_client.cc:523] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.454352 457 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:28.454584 29329 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet ee26e51dcd054057ad08b6db2f4574d8
I20240502 14:00:28.455200 29329 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet d3ae9aa7809c4372aa9d322458baa6b8
I20240502 14:00:28.455488 29329 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet ad03c8bc0c994bc6a9e17c1206f0d8b4
I20240502 14:00:28.455515 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet ee26e51dcd054057ad08b6db2f4574d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:28.455718 32458 tablet_replica.cc:330] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:28.455857 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet ee26e51dcd054057ad08b6db2f4574d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:28.455874 32458 raft_consensus.cc:2229] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.456022 32458 raft_consensus.cc:2258] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.456072 32457 tablet_replica.cc:330] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:28.456221 32457 raft_consensus.cc:2229] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:28.456445 32457 raft_consensus.cc:2258] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.456610 32458 ts_tablet_manager.cc:1875] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.456982 32457 ts_tablet_manager.cc:1875] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.459214 32457 ts_tablet_manager.cc:1888] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:28.459288 32457 log.cc:1196] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee26e51dcd054057ad08b6db2f4574d8
I20240502 14:00:28.459595 32457 ts_tablet_manager.cc:1909] T ee26e51dcd054057ad08b6db2f4574d8 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:28.459707 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ae9aa7809c4372aa9d322458baa6b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:28.459713 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:28.459843 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ae9aa7809c4372aa9d322458baa6b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:28.460044 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ae9aa7809c4372aa9d322458baa6b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:28.460335 32457 tablet_replica.cc:330] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:28.460474 32457 raft_consensus.cc:2229] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.460605 32457 raft_consensus.cc:2258] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.461272 32457 ts_tablet_manager.cc:1875] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.461426 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet d3ae9aa7809c4372aa9d322458baa6b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:28.461426 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:28.461649 32460 tablet_replica.cc:330] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:28.461807 32460 raft_consensus.cc:2229] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:28.461920 29711 tablet_service.cc:1514] Processing DeleteTablet for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:28.462071 32460 raft_consensus.cc:2258] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.462072 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:28.462265 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet ee26e51dcd054057ad08b6db2f4574d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:28.462742 32460 ts_tablet_manager.cc:1875] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.463073 32457 ts_tablet_manager.cc:1888] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.463130 32457 log.cc:1196] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3ae9aa7809c4372aa9d322458baa6b8
I20240502 14:00:28.463294 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet ee26e51dcd054057ad08b6db2f4574d8 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.463382 32457 ts_tablet_manager.cc:1909] T d3ae9aa7809c4372aa9d322458baa6b8 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
W20240502 14:00:28.463497 29315 catalog_manager.cc:4797] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): delete failed for tablet d3ae9aa7809c4372aa9d322458baa6b8 because the tablet was not found. No further retry: Not found: Tablet not found: d3ae9aa7809c4372aa9d322458baa6b8
I20240502 14:00:28.463914 32457 tablet_replica.cc:330] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:28.464125 32457 raft_consensus.cc:2229] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.464314 32457 raft_consensus.cc:2258] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.465557 32581 raft_consensus.cc:2932] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.465916 32458 ts_tablet_manager.cc:1888] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:28.465996 32458 log.cc:1196] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/ee26e51dcd054057ad08b6db2f4574d8
I20240502 14:00:28.466296 32458 ts_tablet_manager.cc:1909] T ee26e51dcd054057ad08b6db2f4574d8 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:28.466969 32458 tablet_replica.cc:330] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:28.467361 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet ee26e51dcd054057ad08b6db2f4574d8 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.467653 32458 raft_consensus.cc:2229] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.467862 32458 raft_consensus.cc:2258] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.467772 29471 raft_consensus.cc:2932] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
W20240502 14:00:28.468508 29313 catalog_manager.cc:4797] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): delete failed for tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 because the tablet was not found. No further retry: Not found: Tablet not found: ad03c8bc0c994bc6a9e17c1206f0d8b4
I20240502 14:00:28.468706 29329 catalog_manager.cc:5484] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.470563 441 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.471194 441 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:28.471849 441 ts_tablet_manager.cc:1367] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.038s user 0.030s sys 0.000s
I20240502 14:00:28.472473 441 raft_consensus.cc:348] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.472676 441 raft_consensus.cc:729] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.473485 29732 raft_consensus.cc:2932] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:28.474089 441 consensus_queue.cc:260] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.475041 441 ts_tablet_manager.cc:1398] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:28.475433 32458 ts_tablet_manager.cc:1875] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.475464 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet d3ae9aa7809c4372aa9d322458baa6b8 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.476003 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-463dddd3473a42a68b488e1c465e997f received from {username='slave'} at 127.22.55.133:57583
I20240502 14:00:28.476086 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-463dddd3473a42a68b488e1c465e997f on tablet 463dddd3473a42a68b488e1c465e997f with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:28.464918 32460 ts_tablet_manager.cc:1888] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.477641 32460 log.cc:1196] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/d3ae9aa7809c4372aa9d322458baa6b8
I20240502 14:00:28.478047 32460 ts_tablet_manager.cc:1909] T d3ae9aa7809c4372aa9d322458baa6b8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:28.478652 32460 tablet_replica.cc:330] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:28.478762 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet d3ae9aa7809c4372aa9d322458baa6b8 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.479182 32460 raft_consensus.cc:2229] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:28.479409 32460 raft_consensus.cc:2258] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.479792 32458 ts_tablet_manager.cc:1888] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.479895 32458 log.cc:1196] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3ae9aa7809c4372aa9d322458baa6b8
I20240502 14:00:28.479957 32457 ts_tablet_manager.cc:1875] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.480269 32458 ts_tablet_manager.cc:1909] T d3ae9aa7809c4372aa9d322458baa6b8 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:28.481127 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet d3ae9aa7809c4372aa9d322458baa6b8 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.481993 32457 ts_tablet_manager.cc:1888] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.482620 32460 ts_tablet_manager.cc:1875] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.487488 32684 raft_consensus.cc:482] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.487614 32684 raft_consensus.cc:504] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.488152 32684 leader_election.cc:288] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:28.489279 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e574c148957f44bf9c2935e955de99aa" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:28.489424 32255 raft_consensus.cc:2454] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:28.489699 31927 leader_election.cc:302] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:28.489917 32684 raft_consensus.cc:2781] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.489975 32684 raft_consensus.cc:482] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.490020 32684 raft_consensus.cc:3037] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.491052 32684 raft_consensus.cc:504] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.491796 32684 leader_election.cc:288] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:28.492401 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e574c148957f44bf9c2935e955de99aa" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:28.492513 32255 raft_consensus.cc:3037] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.493508 32255 raft_consensus.cc:2454] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:28.493887 31927 leader_election.cc:302] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:28.494141 32684 raft_consensus.cc:2781] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.494225 32684 raft_consensus.cc:686] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:28.494345 32684 consensus_queue.cc:237] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.495210 32460 ts_tablet_manager.cc:1888] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.495344 32457 log.cc:1196] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad03c8bc0c994bc6a9e17c1206f0d8b4
I20240502 14:00:28.495461 32460 log.cc:1196] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad03c8bc0c994bc6a9e17c1206f0d8b4
I20240502 14:00:28.495873 32460 ts_tablet_manager.cc:1909] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:28.496160 32457 ts_tablet_manager.cc:1909] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:28.496502 29329 catalog_manager.cc:5484] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.499733 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.499747 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.501637 471 tablet_replica.cc:330] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:28.501828 471 raft_consensus.cc:2229] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.501991 471 raft_consensus.cc:2258] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.502661 471 ts_tablet_manager.cc:1875] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.505713 471 ts_tablet_manager.cc:1888] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.506047 471 log.cc:1196] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/ad03c8bc0c994bc6a9e17c1206f0d8b4
I20240502 14:00:28.506511 471 ts_tablet_manager.cc:1909] T ad03c8bc0c994bc6a9e17c1206f0d8b4 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:28.507310 471 tablet_replica.cc:330] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:28.507411 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet ad03c8bc0c994bc6a9e17c1206f0d8b4 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.507627 471 raft_consensus.cc:2229] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.507759 471 raft_consensus.cc:2258] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.508260 471 ts_tablet_manager.cc:1875] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.510154 471 ts_tablet_manager.cc:1888] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:28.510228 471 log.cc:1196] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee26e51dcd054057ad08b6db2f4574d8
I20240502 14:00:28.510545 471 ts_tablet_manager.cc:1909] T ee26e51dcd054057ad08b6db2f4574d8 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:28.511255 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet ee26e51dcd054057ad08b6db2f4574d8 (table rebalance_test_table_4 [id=18d8b7f0ce2841869a5445e0569e3f29]) successfully deleted
I20240502 14:00:28.511413 457 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.511729 457 tablet_bootstrap.cc:492] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:28.512403 457 ts_tablet_manager.cc:1367] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.058s user 0.037s sys 0.000s
I20240502 14:00:28.512866 457 raft_consensus.cc:348] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.514081 457 raft_consensus.cc:729] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.514539 457 consensus_queue.cc:260] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.515441 441 ts_tablet_manager.cc:902] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:28.515893 441 tablet_copy_client.cc:308] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:28.516657 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 209e6922777b49ddb87a3dadf65d651a from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:57583)
I20240502 14:00:28.516767 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 209e6922777b49ddb87a3dadf65d651a from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:57583: session id = f0e01c4f581843e9ad4ced97f0072dc4-209e6922777b49ddb87a3dadf65d651a
I20240502 14:00:28.516889 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e574c148957f44bf9c2935e955de99aa" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:28.517011 31861 raft_consensus.cc:3037] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.517395 457 ts_tablet_manager.cc:1398] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:28.517791 29753 tablet_copy_source_session.cc:205] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.517979 31861 raft_consensus.cc:2454] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:28.518275 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e574c148957f44bf9c2935e955de99aa" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:28.518402 31860 raft_consensus.cc:2362] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1. Re-sending same reply.
I20240502 14:00:28.519358 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-0867c6a22c2c4efd9a9babff371cc8ba received from {username='slave'} at 127.22.55.135:37007
I20240502 14:00:28.519433 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209e6922777b49ddb87a3dadf65d651a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.519445 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-0867c6a22c2c4efd9a9babff371cc8ba on tablet 0867c6a22c2c4efd9a9babff371cc8ba with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:28.521790 441 tablet_copy_client.cc:783] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.521945 441 tablet_copy_client.cc:647] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.523322 29601 consensus_queue.cc:237] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.524863 29470 raft_consensus.cc:1261] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.525497 29862 raft_consensus.cc:1261] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:28.525593 32561 consensus_queue.cc:1035] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.526497 32561 consensus_queue.cc:1035] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:28.527223 441 tablet_copy_client.cc:523] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.528810 441 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
W20240502 14:00:28.529675 29535 consensus_peers.cc:485] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c35dda6845eb484886eed36e40b9231c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:28.531852 32561 raft_consensus.cc:2932] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.533921 29862 raft_consensus.cc:2932] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.536686 29329 catalog_manager.cc:5484] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.538367 29470 raft_consensus.cc:2932] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:28.546433 499 ts_tablet_manager.cc:902] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:28.549477 499 tablet_copy_client.cc:308] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:28.555398 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet e227b307a0f44a088bf50899d59f2118 from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:34141)
I20240502 14:00:28.555575 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet e227b307a0f44a088bf50899d59f2118 from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:34141: session id = 8973f2527acb48f5ad36589652de73e1-e227b307a0f44a088bf50899d59f2118
I20240502 14:00:28.556973 29884 tablet_copy_source_session.cc:205] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.559012 499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e227b307a0f44a088bf50899d59f2118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.561614 499 tablet_copy_client.cc:783] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.561800 499 tablet_copy_client.cc:647] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.564847 499 tablet_copy_client.cc:523] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.566399 499 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.569459 441 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.569723 441 tablet_bootstrap.cc:492] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:28.569936 441 ts_tablet_manager.cc:1367] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.000s
I20240502 14:00:28.570353 441 raft_consensus.cc:348] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.570459 441 raft_consensus.cc:729] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.570624 441 consensus_queue.cc:260] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.571004 441 ts_tablet_manager.cc:1398] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:00:28.571545 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-209e6922777b49ddb87a3dadf65d651a received from {username='slave'} at 127.22.55.133:57583
I20240502 14:00:28.571640 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-209e6922777b49ddb87a3dadf65d651a on tablet 209e6922777b49ddb87a3dadf65d651a with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:28.574311 518 ts_tablet_manager.cc:902] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:28.575131 518 tablet_copy_client.cc:308] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:28.578917 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 36daa6b40d8a48b8956b67a97833eff5 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:42159)
I20240502 14:00:28.579026 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 36daa6b40d8a48b8956b67a97833eff5 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:42159: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-36daa6b40d8a48b8956b67a97833eff5
I20240502 14:00:28.579759 29884 tablet_copy_source_session.cc:205] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.580977 518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36daa6b40d8a48b8956b67a97833eff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.582984 518 tablet_copy_client.cc:783] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.583194 518 tablet_copy_client.cc:647] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.584704 518 tablet_copy_client.cc:523] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.586815 518 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:28.592000 457 ts_tablet_manager.cc:902] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:28.592490 457 tablet_copy_client.cc:308] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:28.593017 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 253ea421888249d180928ba0aef28ddd from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37007)
I20240502 14:00:28.593127 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 253ea421888249d180928ba0aef28ddd from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37007: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-253ea421888249d180928ba0aef28ddd
I20240502 14:00:28.594106 29622 tablet_copy_source_session.cc:205] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.595248 457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253ea421888249d180928ba0aef28ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.596393 499 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.596701 499 tablet_bootstrap.cc:492] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:28.596902 499 ts_tablet_manager.cc:1367] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.003s
I20240502 14:00:28.597087 457 tablet_copy_client.cc:783] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.597254 457 tablet_copy_client.cc:647] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.597357 499 raft_consensus.cc:348] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.597497 499 raft_consensus.cc:729] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.597652 499 consensus_queue.cc:260] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.597915 499 ts_tablet_manager.cc:1398] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:00:28.598515 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-e227b307a0f44a088bf50899d59f2118 received from {username='slave'} at 127.22.55.136:34141
I20240502 14:00:28.598623 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-e227b307a0f44a088bf50899d59f2118 on tablet e227b307a0f44a088bf50899d59f2118 with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:28.598431 457 tablet_copy_client.cc:523] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.603020 457 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:28.620373 518 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.620769 518 tablet_bootstrap.cc:492] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:28.621021 518 ts_tablet_manager.cc:1367] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20240502 14:00:28.621520 518 raft_consensus.cc:348] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.621660 518 raft_consensus.cc:729] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.621836 518 consensus_queue.cc:260] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.622167 518 ts_tablet_manager.cc:1398] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.622710 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-36daa6b40d8a48b8956b67a97833eff5 received from {username='slave'} at 127.22.55.134:42159
I20240502 14:00:28.622802 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-36daa6b40d8a48b8956b67a97833eff5 on tablet 36daa6b40d8a48b8956b67a97833eff5 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:28.633600 457 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.633980 457 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:28.634181 457 ts_tablet_manager.cc:1367] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.006s
I20240502 14:00:28.634646 457 raft_consensus.cc:348] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.634819 457 raft_consensus.cc:729] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.635010 457 consensus_queue.cc:260] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.636025 457 ts_tablet_manager.cc:1398] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:28.636595 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-253ea421888249d180928ba0aef28ddd received from {username='slave'} at 127.22.55.135:37007
I20240502 14:00:28.636678 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-253ea421888249d180928ba0aef28ddd on tablet 253ea421888249d180928ba0aef28ddd with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:28.685566 31659 raft_consensus.cc:482] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.685694 31659 raft_consensus.cc:504] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.686098 31659 leader_election.cc:288] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:28.686513 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cff896decf14fca8307ff326a8f64ef" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:28.686525 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cff896decf14fca8307ff326a8f64ef" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:28.686653 29862 raft_consensus.cc:2454] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:28.686666 32255 raft_consensus.cc:2454] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:28.686962 29536 leader_election.cc:302] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:28.687134 31659 raft_consensus.cc:2781] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.687202 31659 raft_consensus.cc:482] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.687242 31659 raft_consensus.cc:3037] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.688249 31659 raft_consensus.cc:504] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.688526 31659 leader_election.cc:288] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:28.688870 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cff896decf14fca8307ff326a8f64ef" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:28.688968 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cff896decf14fca8307ff326a8f64ef" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:28.689011 29862 raft_consensus.cc:3037] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.689103 32255 raft_consensus.cc:3037] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.689836 32255 raft_consensus.cc:2454] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:28.689873 29862 raft_consensus.cc:2454] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:28.690110 29537 leader_election.cc:302] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:28.690279 31659 raft_consensus.cc:2781] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.690387 31659 raft_consensus.cc:686] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:28.690519 31659 consensus_queue.cc:237] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.692561 29328 catalog_manager.cc:5484] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:28.694164 499 ts_tablet_manager.cc:902] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:28.694672 499 tablet_copy_client.cc:308] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:28.695164 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet f4697431776e44a1907cf4b0865414da from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:34141)
I20240502 14:00:28.695266 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet f4697431776e44a1907cf4b0865414da from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:34141: session id = 8973f2527acb48f5ad36589652de73e1-f4697431776e44a1907cf4b0865414da
I20240502 14:00:28.696135 29884 tablet_copy_source_session.cc:205] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.702466 499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4697431776e44a1907cf4b0865414da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.704829 31659 consensus_queue.cc:1035] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.705453 499 tablet_copy_client.cc:783] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.705636 499 tablet_copy_client.cc:647] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.706673 499 tablet_copy_client.cc:523] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.707958 499 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.716442 441 ts_tablet_manager.cc:902] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:28.716853 441 tablet_copy_client.cc:308] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:28.720566 29844 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:28.721381 31841 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.722198 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:28.724150 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet 6767168f6f2b4e15aab7e64b0eab57c3 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:34855)
I20240502 14:00:28.724280 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet 6767168f6f2b4e15aab7e64b0eab57c3 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:34855: session id = f0e01c4f581843e9ad4ced97f0072dc4-6767168f6f2b4e15aab7e64b0eab57c3
I20240502 14:00:28.725752 29491 tablet_copy_source_session.cc:205] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.726941 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6767168f6f2b4e15aab7e64b0eab57c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.727927 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:28.729246 441 tablet_copy_client.cc:783] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.729394 441 tablet_copy_client.cc:647] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.729615 32233 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.743228 32561 consensus_queue.cc:1035] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.750802 31971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.753455 32103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.762900 441 tablet_copy_client.cc:523] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.764465 457 ts_tablet_manager.cc:902] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:28.764798 441 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:28.764994 457 tablet_copy_client.cc:308] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:28.771412 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet a3bf63852df44cf1b762b7a981428c97 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37007)
I20240502 14:00:28.771555 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet a3bf63852df44cf1b762b7a981428c97 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37007: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-a3bf63852df44cf1b762b7a981428c97
I20240502 14:00:28.772686 29622 tablet_copy_source_session.cc:205] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.773952 457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf63852df44cf1b762b7a981428c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.776108 457 tablet_copy_client.cc:783] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.776243 457 tablet_copy_client.cc:647] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.778666 29580 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:28.782685 518 ts_tablet_manager.cc:902] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:28.783166 518 tablet_copy_client.cc:308] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:28.783290 582 ts_tablet_manager.cc:902] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:28.783674 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet f3517a9ff83644ccb5cd4c0ebbd6e929 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:42159)
I20240502 14:00:28.783785 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet f3517a9ff83644ccb5cd4c0ebbd6e929 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:42159: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-f3517a9ff83644ccb5cd4c0ebbd6e929
I20240502 14:00:28.783804 582 tablet_copy_client.cc:308] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:28.784821 29884 tablet_copy_source_session.cc:205] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.785373 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet 59235c5f3daf4ff1afeab83efe018405 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:34855)
I20240502 14:00:28.785478 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet 59235c5f3daf4ff1afeab83efe018405 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:34855: session id = f0e01c4f581843e9ad4ced97f0072dc4-59235c5f3daf4ff1afeab83efe018405
I20240502 14:00:28.785944 457 tablet_copy_client.cc:523] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.786123 518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3517a9ff83644ccb5cd4c0ebbd6e929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.786566 29491 tablet_copy_source_session.cc:205] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.788156 457 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:28.788257 518 tablet_copy_client.cc:783] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.788409 518 tablet_copy_client.cc:647] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.789063 582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59235c5f3daf4ff1afeab83efe018405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.791944 582 tablet_copy_client.cc:783] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.792117 582 tablet_copy_client.cc:647] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.798267 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:28.803432 32255 raft_consensus.cc:1261] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.803952 32561 consensus_queue.cc:1035] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.805326 29862 raft_consensus.cc:1261] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.807724 518 tablet_copy_client.cc:523] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.808430 32602 consensus_queue.cc:1035] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.809165 518 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:28.810560 499 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.810842 499 tablet_bootstrap.cc:492] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:28.811163 499 ts_tablet_manager.cc:1367] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.103s user 0.022s sys 0.004s
I20240502 14:00:28.811659 499 raft_consensus.cc:348] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.811775 499 raft_consensus.cc:729] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.811918 499 consensus_queue.cc:260] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.812997 29329 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet 1cff896decf14fca8307ff326a8f64ef (table rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457]). Expected version 1 got 0
I20240502 14:00:28.812275 499 ts_tablet_manager.cc:1398] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.814049 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-f4697431776e44a1907cf4b0865414da received from {username='slave'} at 127.22.55.136:34141
I20240502 14:00:28.814139 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-f4697431776e44a1907cf4b0865414da on tablet f4697431776e44a1907cf4b0865414da with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:28.815418 29329 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 1cff896decf14fca8307ff326a8f64ef (table rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457]). Expected version 1 got 0
I20240502 14:00:28.816040 31860 raft_consensus.cc:1203] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:28.829509 457 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.829761 582 tablet_copy_client.cc:523] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.829901 457 tablet_bootstrap.cc:492] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:28.830088 457 ts_tablet_manager.cc:1367] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.042s user 0.026s sys 0.000s
I20240502 14:00:28.830582 457 raft_consensus.cc:348] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.830701 457 raft_consensus.cc:729] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.830828 457 consensus_queue.cc:260] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:28.831115 457 ts_tablet_manager.cc:1398] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.831312 582 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:28.831724 499 ts_tablet_manager.cc:902] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:28.832023 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-a3bf63852df44cf1b762b7a981428c97 received from {username='slave'} at 127.22.55.135:37007
I20240502 14:00:28.832110 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-a3bf63852df44cf1b762b7a981428c97 on tablet a3bf63852df44cf1b762b7a981428c97 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:28.833200 29329 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet 1cff896decf14fca8307ff326a8f64ef (table rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457]). Expected version 1 got 0
I20240502 14:00:28.834793 29329 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 1cff896decf14fca8307ff326a8f64ef (table rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457]). Expected version 1 got 0
I20240502 14:00:28.836870 499 tablet_copy_client.cc:308] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:28.837118 605 ts_tablet_manager.cc:902] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:28.837878 605 tablet_copy_client.cc:308] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:28.838351 607 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:28.838928 606 ts_tablet_manager.cc:902] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:28.839521 606 tablet_copy_client.cc:308] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:28.839910 608 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:28.840742 608 tablet.cc:1620] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:28.841645 607 tablet.cc:1620] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:28.841791 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet ba65f51fa3f64a479e1e62042c1ca64e from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57283)
I20240502 14:00:28.841871 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet ba65f51fa3f64a479e1e62042c1ca64e from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57283: session id = 8973f2527acb48f5ad36589652de73e1-ba65f51fa3f64a479e1e62042c1ca64e
I20240502 14:00:28.842511 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet b2c754e60c1d4f6f95a807e9644bd21f from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57511)
I20240502 14:00:28.842608 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet b2c754e60c1d4f6f95a807e9644bd21f from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57511: session id = 8973f2527acb48f5ad36589652de73e1-b2c754e60c1d4f6f95a807e9644bd21f
I20240502 14:00:28.842818 29622 tablet_copy_source_session.cc:205] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.843662 29491 tablet_copy_source_session.cc:205] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.844514 499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba65f51fa3f64a479e1e62042c1ca64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.846225 606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c754e60c1d4f6f95a807e9644bd21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.846655 499 tablet_copy_client.cc:783] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.846787 499 tablet_copy_client.cc:647] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.848875 609 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:28.849315 609 tablet.cc:1620] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:28.849903 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 9bdf925a9c4d456fb3b4ef448a2154c6 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:50019)
I20240502 14:00:28.850008 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 9bdf925a9c4d456fb3b4ef448a2154c6 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:50019: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-9bdf925a9c4d456fb3b4ef448a2154c6
I20240502 14:00:28.851120 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457] alter complete (version 1)
I20240502 14:00:28.851150 29753 tablet_copy_source_session.cc:205] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:28.851652 32752 consensus_queue.cc:1035] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.852203 605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdf925a9c4d456fb3b4ef448a2154c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.852725 606 tablet_copy_client.cc:783] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.853603 606 tablet_copy_client.cc:647] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.854766 605 tablet_copy_client.cc:783] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:28.855273 605 tablet_copy_client.cc:647] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:28.855909 441 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.856340 605 tablet_copy_client.cc:523] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.856513 499 tablet_copy_client.cc:523] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.857595 605 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:28.858325 499 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.858697 606 tablet_copy_client.cc:523] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:28.859953 606 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:28.866014 32749 consensus_queue.cc:1035] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.870316 441 tablet_bootstrap.cc:492] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:28.870647 441 ts_tablet_manager.cc:1367] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.106s user 0.028s sys 0.000s
I20240502 14:00:28.872339 32684 consensus_queue.cc:1035] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.873148 441 raft_consensus.cc:348] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.875548 441 raft_consensus.cc:729] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.875799 441 consensus_queue.cc:260] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.883428 441 ts_tablet_manager.cc:1398] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240502 14:00:28.884294 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-6767168f6f2b4e15aab7e64b0eab57c3 received from {username='slave'} at 127.22.55.133:34855
I20240502 14:00:28.884403 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-6767168f6f2b4e15aab7e64b0eab57c3 on tablet 6767168f6f2b4e15aab7e64b0eab57c3 with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:28.899662 582 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.900095 518 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.900146 582 tablet_bootstrap.cc:492] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:28.900336 582 ts_tablet_manager.cc:1367] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.069s user 0.025s sys 0.002s
I20240502 14:00:28.900868 582 raft_consensus.cc:348] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.901016 582 raft_consensus.cc:729] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.901191 582 consensus_queue.cc:260] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:28.901726 582 ts_tablet_manager.cc:1398] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.902451 32542 consensus_queue.cc:1035] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.902683 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-59235c5f3daf4ff1afeab83efe018405 received from {username='slave'} at 127.22.55.133:34855
I20240502 14:00:28.902760 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-59235c5f3daf4ff1afeab83efe018405 on tablet 59235c5f3daf4ff1afeab83efe018405 with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:28.903225 518 tablet_bootstrap.cc:492] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:28.903406 499 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.903544 518 ts_tablet_manager.cc:1367] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.094s user 0.028s sys 0.000s
I20240502 14:00:28.903997 518 raft_consensus.cc:348] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.904139 518 raft_consensus.cc:729] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.904320 518 consensus_queue.cc:260] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.904703 518 ts_tablet_manager.cc:1398] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.905262 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-f3517a9ff83644ccb5cd4c0ebbd6e929 received from {username='slave'} at 127.22.55.134:42159
I20240502 14:00:28.905347 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-f3517a9ff83644ccb5cd4c0ebbd6e929 on tablet f3517a9ff83644ccb5cd4c0ebbd6e929 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:28.915212 606 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.931071 499 tablet_bootstrap.cc:492] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:28.931320 499 ts_tablet_manager.cc:1367] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.073s user 0.020s sys 0.007s
I20240502 14:00:28.931329 606 tablet_bootstrap.cc:492] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:28.931685 606 ts_tablet_manager.cc:1367] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.072s user 0.025s sys 0.001s
I20240502 14:00:28.931834 499 raft_consensus.cc:348] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.931942 499 raft_consensus.cc:729] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.932010 606 raft_consensus.cc:348] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.932094 606 raft_consensus.cc:729] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.932229 606 consensus_queue.cc:260] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.932536 606 ts_tablet_manager.cc:1398] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.932093 499 consensus_queue.cc:260] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:28.933903 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-b2c754e60c1d4f6f95a807e9644bd21f received from {username='slave'} at 127.22.55.136:57511
I20240502 14:00:28.933988 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-b2c754e60c1d4f6f95a807e9644bd21f on tablet b2c754e60c1d4f6f95a807e9644bd21f with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:28.935070 499 ts_tablet_manager.cc:1398] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:28.943022 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-ba65f51fa3f64a479e1e62042c1ca64e received from {username='slave'} at 127.22.55.136:57283
I20240502 14:00:28.943197 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-ba65f51fa3f64a479e1e62042c1ca64e on tablet ba65f51fa3f64a479e1e62042c1ca64e with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:28.958684 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:28.960811 605 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.961331 605 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:28.961575 605 ts_tablet_manager.cc:1367] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.104s user 0.028s sys 0.000s
I20240502 14:00:28.963256 605 raft_consensus.cc:348] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.964264 605 raft_consensus.cc:729] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.964498 605 consensus_queue.cc:260] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:28.964639 607 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:28.964951 607 tablet.cc:1620] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:28.965348 608 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:28.965510 605 ts_tablet_manager.cc:1398] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:28.965540 609 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:28.965641 608 tablet.cc:1620] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:28.965792 609 tablet.cc:1620] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:28.965981 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-9bdf925a9c4d456fb3b4ef448a2154c6 received from {username='slave'} at 127.22.55.134:50019
I20240502 14:00:28.966061 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-9bdf925a9c4d456fb3b4ef448a2154c6 on tablet 9bdf925a9c4d456fb3b4ef448a2154c6 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:28.966967 29329 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 14:00:28.967161 29329 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 14:00:28.973042 29329 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet a3bf63852df44cf1b762b7a981428c97
I20240502 14:00:28.973434 29329 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 74be4a1291264d3090849cb66bbca085
I20240502 14:00:28.973600 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=506cb4e787674dfcac287afd5f15e457] alter complete (version 2)
I20240502 14:00:28.973672 29329 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet 59235c5f3daf4ff1afeab83efe018405
I20240502 14:00:28.974447 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf63852df44cf1b762b7a981428c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:28.974443 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 74be4a1291264d3090849cb66bbca085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:28.974573 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 59235c5f3daf4ff1afeab83efe018405 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:28.974591 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet 74be4a1291264d3090849cb66bbca085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:28.974619 32460 tablet_replica.cc:330] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:28.974740 471 tablet_replica.cc:330] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:28.974800 32460 raft_consensus.cc:2229] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.974910 471 raft_consensus.cc:2229] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.974972 32460 raft_consensus.cc:2258] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.975044 471 raft_consensus.cc:2258] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.975049 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf63852df44cf1b762b7a981428c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:28.975150 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf63852df44cf1b762b7a981428c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:28.975215 32458 tablet_replica.cc:330] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:28.975311 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 74be4a1291264d3090849cb66bbca085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:28.975355 32458 raft_consensus.cc:2229] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:28.975418 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet 59235c5f3daf4ff1afeab83efe018405 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:28.975529 32457 tablet_replica.cc:330] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:28.975569 471 ts_tablet_manager.cc:1875] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.975608 32458 raft_consensus.cc:2258] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.975648 32457 raft_consensus.cc:2229] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.975679 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 59235c5f3daf4ff1afeab83efe018405 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:28.975792 32457 raft_consensus.cc:2258] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.976817 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet 59235c5f3daf4ff1afeab83efe018405 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:28.976923 32457 ts_tablet_manager.cc:1875] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.977770 471 ts_tablet_manager.cc:1888] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:28.977864 471 log.cc:1196] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/74be4a1291264d3090849cb66bbca085
I20240502 14:00:28.978179 471 ts_tablet_manager.cc:1909] T 74be4a1291264d3090849cb66bbca085 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:28.978891 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 74be4a1291264d3090849cb66bbca085 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.979058 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf63852df44cf1b762b7a981428c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:28 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:28.979386 673 tablet_replica.cc:330] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:28.979585 673 raft_consensus.cc:2229] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.979758 673 raft_consensus.cc:2258] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.979871 32458 ts_tablet_manager.cc:1875] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.979920 32457 ts_tablet_manager.cc:1888] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.980053 32457 log.cc:1196] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/a3bf63852df44cf1b762b7a981428c97
I20240502 14:00:28.980162 32460 ts_tablet_manager.cc:1875] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.980360 32457 ts_tablet_manager.cc:1909] T a3bf63852df44cf1b762b7a981428c97 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:28.980518 673 ts_tablet_manager.cc:1875] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.981047 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet a3bf63852df44cf1b762b7a981428c97 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.981320 32457 tablet_replica.cc:330] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:28.981451 32457 raft_consensus.cc:2229] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:28.981638 32457 raft_consensus.cc:2258] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.982064 32458 ts_tablet_manager.cc:1888] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.982093 32457 ts_tablet_manager.cc:1875] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.982127 32458 log.cc:1196] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3bf63852df44cf1b762b7a981428c97
I20240502 14:00:28.982332 32460 ts_tablet_manager.cc:1888] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.982398 32460 log.cc:1196] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/a3bf63852df44cf1b762b7a981428c97
I20240502 14:00:28.982439 32458 ts_tablet_manager.cc:1909] T a3bf63852df44cf1b762b7a981428c97 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:28.982645 32460 ts_tablet_manager.cc:1909] T a3bf63852df44cf1b762b7a981428c97 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:28.982883 673 ts_tablet_manager.cc:1888] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.982949 673 log.cc:1196] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/a3bf63852df44cf1b762b7a981428c97
I20240502 14:00:28.983144 32458 tablet_replica.cc:330] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:28.983232 673 ts_tablet_manager.cc:1909] T a3bf63852df44cf1b762b7a981428c97 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:28.983278 32458 raft_consensus.cc:2229] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.983294 32460 tablet_replica.cc:330] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:28.983435 32458 raft_consensus.cc:2258] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.983438 32460 raft_consensus.cc:2229] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.983561 32460 raft_consensus.cc:2258] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.983892 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet a3bf63852df44cf1b762b7a981428c97 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.984165 32457 ts_tablet_manager.cc:1888] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:28.984238 32457 log.cc:1196] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/74be4a1291264d3090849cb66bbca085
I20240502 14:00:28.984313 32460 ts_tablet_manager.cc:1875] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.984505 32457 ts_tablet_manager.cc:1909] T 74be4a1291264d3090849cb66bbca085 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:28.984686 32458 ts_tablet_manager.cc:1875] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.985107 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 74be4a1291264d3090849cb66bbca085 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.986392 32460 ts_tablet_manager.cc:1888] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.986513 32460 log.cc:1196] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/59235c5f3daf4ff1afeab83efe018405
I20240502 14:00:28.986793 32457 tablet_replica.cc:330] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:28.986827 32460 ts_tablet_manager.cc:1909] T 59235c5f3daf4ff1afeab83efe018405 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:28.986986 32457 raft_consensus.cc:2229] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:28.987138 32458 ts_tablet_manager.cc:1888] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:28.987259 32458 log.cc:1196] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/59235c5f3daf4ff1afeab83efe018405
I20240502 14:00:28.987524 32460 tablet_replica.cc:330] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:28.987638 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 59235c5f3daf4ff1afeab83efe018405 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.987692 32460 raft_consensus.cc:2229] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.987910 32460 raft_consensus.cc:2258] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.988466 32457 raft_consensus.cc:2258] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.988777 32458 ts_tablet_manager.cc:1909] T 59235c5f3daf4ff1afeab83efe018405 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:28.989269 32457 ts_tablet_manager.cc:1875] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.989717 32460 ts_tablet_manager.cc:1875] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:28.993002 32460 ts_tablet_manager.cc:1888] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:28.993161 32460 log.cc:1196] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/74be4a1291264d3090849cb66bbca085
I20240502 14:00:28.993503 32460 ts_tablet_manager.cc:1909] T 74be4a1291264d3090849cb66bbca085 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:28.994871 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 74be4a1291264d3090849cb66bbca085 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.995231 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet a3bf63852df44cf1b762b7a981428c97 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.995344 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 59235c5f3daf4ff1afeab83efe018405 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:28.995882 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet a3bf63852df44cf1b762b7a981428c97 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:29.001749 31860 raft_consensus.cc:1203] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.007239 32457 ts_tablet_manager.cc:1888] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.007335 32457 log.cc:1196] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/59235c5f3daf4ff1afeab83efe018405
I20240502 14:00:29.007665 32457 ts_tablet_manager.cc:1909] T 59235c5f3daf4ff1afeab83efe018405 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:29.008553 672 tablet_replica.cc:330] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:29.008742 672 raft_consensus.cc:2229] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.008900 672 raft_consensus.cc:2258] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.009025 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 59235c5f3daf4ff1afeab83efe018405 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:29.009645 672 ts_tablet_manager.cc:1875] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.011096 457 ts_tablet_manager.cc:902] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:29.011554 457 tablet_copy_client.cc:308] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:29.012593 672 ts_tablet_manager.cc:1888] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.012665 672 log.cc:1196] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/59235c5f3daf4ff1afeab83efe018405
I20240502 14:00:29.012955 672 ts_tablet_manager.cc:1909] T 59235c5f3daf4ff1afeab83efe018405 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:29.014626 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 59235c5f3daf4ff1afeab83efe018405 (table rebalance_test_table_6 [id=9f5a41bba20448dcba9be999e13b0f22]) successfully deleted
I20240502 14:00:29.023464 29842 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:29.024897 29580 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:29.026655 31971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.026815 31841 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.029078 32254 raft_consensus.cc:1203] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.032199 29450 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.032531 32119 raft_consensus.cc:1203] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.040521 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.048863 32119 raft_consensus.cc:1203] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.051998 31991 raft_consensus.cc:1203] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.061307 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 7ebf61381a92430caa0c38cc04d14cb6 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37167)
I20240502 14:00:29.061455 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 7ebf61381a92430caa0c38cc04d14cb6 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37167: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-7ebf61381a92430caa0c38cc04d14cb6
I20240502 14:00:29.062441 29884 tablet_copy_source_session.cc:205] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:29.064309 32103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.065325 457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ebf61381a92430caa0c38cc04d14cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.067605 457 tablet_copy_client.cc:783] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:29.067760 457 tablet_copy_client.cc:647] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:29.069124 457 tablet_copy_client.cc:523] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:29.069331 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:29.070408 457 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:29.078339 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 6d3a84c1d004412db381854147330dfc (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.078656 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d3a84c1d004412db381854147330dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.079070 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 6d3a84c1d004412db381854147330dfc (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.079259 32233 tablet_service.cc:1467] Processing CreateTablet for tablet a1116d9f165b488796ab466f9a866471 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.079362 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d3a84c1d004412db381854147330dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.079519 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116d9f165b488796ab466f9a866471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.081051 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 70367c7c718d4d9195f0af7e69bdf2ec (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.081277 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70367c7c718d4d9195f0af7e69bdf2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.081650 32103 tablet_service.cc:1467] Processing CreateTablet for tablet a1116d9f165b488796ab466f9a866471 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.081861 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116d9f165b488796ab466f9a866471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.081892 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 70367c7c718d4d9195f0af7e69bdf2ec (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.082113 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70367c7c718d4d9195f0af7e69bdf2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.083797 31971 tablet_service.cc:1467] Processing CreateTablet for tablet a1116d9f165b488796ab466f9a866471 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.084053 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116d9f165b488796ab466f9a866471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.084139 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 70367c7c718d4d9195f0af7e69bdf2ec (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.084349 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70367c7c718d4d9195f0af7e69bdf2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.086455 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 6d3a84c1d004412db381854147330dfc (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.086674 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d3a84c1d004412db381854147330dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.090265 693 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:29.090543 692 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:29.090567 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:29.091691 693 tablet_bootstrap.cc:654] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.092646 692 tablet_bootstrap.cc:654] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.095402 695 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:29.096335 694 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:29.097278 694 tablet_bootstrap.cc:654] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.098452 694 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:29.099146 694 ts_tablet_manager.cc:1367] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:29.099634 694 raft_consensus.cc:348] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.099862 694 raft_consensus.cc:374] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.099937 694 raft_consensus.cc:729] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.100128 694 consensus_queue.cc:260] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.101333 457 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:29.101810 457 tablet_bootstrap.cc:492] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:29.102008 457 ts_tablet_manager.cc:1367] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.000s
I20240502 14:00:29.102191 694 ts_tablet_manager.cc:1398] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:29.102644 457 raft_consensus.cc:348] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.102819 457 raft_consensus.cc:729] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.103382 457 consensus_queue.cc:260] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.103718 457 ts_tablet_manager.cc:1398] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:29.104552 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-7ebf61381a92430caa0c38cc04d14cb6 received from {username='slave'} at 127.22.55.135:37167
I20240502 14:00:29.104645 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-7ebf61381a92430caa0c38cc04d14cb6 on tablet 7ebf61381a92430caa0c38cc04d14cb6 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:29.104782 695 tablet_bootstrap.cc:654] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.097970 31860 raft_consensus.cc:1203] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.106177 607 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:29.108013 609 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:29.109759 608 tablet.cc:1722] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:29.111014 694 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:29.111311 692 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:29.111409 692 ts_tablet_manager.cc:1367] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240502 14:00:29.112511 694 tablet_bootstrap.cc:654] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.112718 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=506cb4e787674dfcac287afd5f15e457] alter complete (version 3)
I20240502 14:00:29.113353 692 raft_consensus.cc:348] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.113524 692 raft_consensus.cc:374] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.113565 692 raft_consensus.cc:729] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.113821 692 consensus_queue.cc:260] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.114213 692 ts_tablet_manager.cc:1398] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:29.114370 692 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:29.115206 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.115417 692 tablet_bootstrap.cc:654] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.117260 692 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:29.117373 692 ts_tablet_manager.cc:1367] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:29.117434 693 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:29.117928 692 raft_consensus.cc:348] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.118062 692 raft_consensus.cc:374] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.118126 692 raft_consensus.cc:729] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.118283 692 consensus_queue.cc:260] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.118571 692 ts_tablet_manager.cc:1398] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.119393 693 ts_tablet_manager.cc:1367] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:00:29.121076 693 raft_consensus.cc:348] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.121606 695 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:29.121704 695 ts_tablet_manager.cc:1367] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240502 14:00:29.122074 695 raft_consensus.cc:348] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.122176 695 raft_consensus.cc:374] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.122220 695 raft_consensus.cc:729] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.122357 693 raft_consensus.cc:374] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.122429 693 raft_consensus.cc:729] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.122951 693 consensus_queue.cc:260] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.123525 693 ts_tablet_manager.cc:1398] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:29.123744 693 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:29.125619 693 tablet_bootstrap.cc:654] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.126782 695 consensus_queue.cc:260] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.127231 695 ts_tablet_manager.cc:1398] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:29.127699 693 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:29.127940 695 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:29.129541 695 tablet_bootstrap.cc:654] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.131121 695 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:29.131258 695 ts_tablet_manager.cc:1367] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:00:29.131335 693 ts_tablet_manager.cc:1367] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:29.131821 693 raft_consensus.cc:348] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.131821 695 raft_consensus.cc:348] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.131937 695 raft_consensus.cc:374] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.131937 693 raft_consensus.cc:374] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.131973 693 raft_consensus.cc:729] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.131973 695 raft_consensus.cc:729] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.132119 695 consensus_queue.cc:260] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.132143 693 consensus_queue.cc:260] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.132489 695 ts_tablet_manager.cc:1398] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.134078 693 ts_tablet_manager.cc:1398] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:29.135402 693 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:29.136198 693 tablet_bootstrap.cc:654] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.137792 693 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:29.137861 693 ts_tablet_manager.cc:1367] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:29.138315 693 raft_consensus.cc:348] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.138454 693 raft_consensus.cc:374] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.138507 693 raft_consensus.cc:729] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.138659 693 consensus_queue.cc:260] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.139016 693 ts_tablet_manager.cc:1398] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.141813 582 ts_tablet_manager.cc:902] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:29.142345 582 tablet_copy_client.cc:308] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:29.147482 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet c35dda6845eb484886eed36e40b9231c from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:49783)
I20240502 14:00:29.147553 694 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:29.147584 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet c35dda6845eb484886eed36e40b9231c from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:49783: session id = f0e01c4f581843e9ad4ced97f0072dc4-c35dda6845eb484886eed36e40b9231c
I20240502 14:00:29.147641 694 ts_tablet_manager.cc:1367] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240502 14:00:29.148118 694 raft_consensus.cc:348] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.148232 694 raft_consensus.cc:374] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.148259 694 raft_consensus.cc:729] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.148404 29622 tablet_copy_source_session.cc:205] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:29.148384 694 consensus_queue.cc:260] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.148626 694 ts_tablet_manager.cc:1398] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.149570 582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35dda6845eb484886eed36e40b9231c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.151867 582 tablet_copy_client.cc:783] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:29.152001 582 tablet_copy_client.cc:647] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:29.153201 582 tablet_copy_client.cc:523] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:29.154695 582 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:29.180595 582 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:29.180894 582 tablet_bootstrap.cc:492] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:29.181074 582 ts_tablet_manager.cc:1367] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240502 14:00:29.181465 582 raft_consensus.cc:348] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.181578 582 raft_consensus.cc:729] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.181707 582 consensus_queue.cc:260] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.181942 582 ts_tablet_manager.cc:1398] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:29.182930 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-c35dda6845eb484886eed36e40b9231c received from {username='slave'} at 127.22.55.133:49783
I20240502 14:00:29.183028 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-c35dda6845eb484886eed36e40b9231c on tablet c35dda6845eb484886eed36e40b9231c with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:29.185634 32749 raft_consensus.cc:482] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.185735 32749 raft_consensus.cc:504] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.186143 32749 leader_election.cc:288] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:29.186668 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3a84c1d004412db381854147330dfc" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:29.186797 31991 raft_consensus.cc:2454] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:29.187121 32056 leader_election.cc:302] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.187733 32749 raft_consensus.cc:2781] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.187789 32749 raft_consensus.cc:482] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.187826 32749 raft_consensus.cc:3037] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.187783 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3a84c1d004412db381854147330dfc" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:29.187904 31860 raft_consensus.cc:2454] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:29.188917 32749 raft_consensus.cc:504] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.189282 32749 leader_election.cc:288] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:29.189654 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3a84c1d004412db381854147330dfc" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:29.189770 31860 raft_consensus.cc:3037] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.190788 31860 raft_consensus.cc:2454] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:29.191102 32056 leader_election.cc:302] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.191359 32749 raft_consensus.cc:2781] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.191449 32749 raft_consensus.cc:686] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:29.191587 32749 consensus_queue.cc:237] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:29.192066 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3a84c1d004412db381854147330dfc" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:29.192181 31991 raft_consensus.cc:3037] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.193221 31991 raft_consensus.cc:2454] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:29.193512 29323 catalog_manager.cc:5484] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:29.199509 32749 raft_consensus.cc:482] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.199604 32749 raft_consensus.cc:504] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.199887 31991 raft_consensus.cc:1203] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.200341 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70367c7c718d4d9195f0af7e69bdf2ec" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:29.200505 31860 raft_consensus.cc:2454] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:29.201287 32056 leader_election.cc:302] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.201339 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70367c7c718d4d9195f0af7e69bdf2ec" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:29.201460 32752 raft_consensus.cc:2781] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.201490 32254 raft_consensus.cc:2454] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:29.201510 32752 raft_consensus.cc:482] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.201556 32752 raft_consensus.cc:3037] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.202644 32752 raft_consensus.cc:504] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.203013 32752 leader_election.cc:288] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:29.203513 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70367c7c718d4d9195f0af7e69bdf2ec" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:29.203625 32254 raft_consensus.cc:3037] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.204309 32749 leader_election.cc:288] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:29.204728 32254 raft_consensus.cc:2454] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:29.204831 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70367c7c718d4d9195f0af7e69bdf2ec" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:29.204946 31860 raft_consensus.cc:3037] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.205714 32058 leader_election.cc:302] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.205873 32749 raft_consensus.cc:2781] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.205955 31860 raft_consensus.cc:2454] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:29.206099 32749 raft_consensus.cc:686] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:29.206563 32749 consensus_queue.cc:237] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.209092 29325 catalog_manager.cc:5484] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:29.214514 29327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:29.214627 29327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:29.216660 29327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 1cff896decf14fca8307ff326a8f64ef
I20240502 14:00:29.217226 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cff896decf14fca8307ff326a8f64ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:29.217439 32458 tablet_replica.cc:330] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:29.217590 32458 raft_consensus.cc:2229] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:29.217792 32458 raft_consensus.cc:2258] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.217967 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cff896decf14fca8307ff326a8f64ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:29.218178 32460 tablet_replica.cc:330] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:29.218336 32460 raft_consensus.cc:2229] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.218468 32460 raft_consensus.cc:2258] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.218884 32458 ts_tablet_manager.cc:1875] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.219002 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cff896decf14fca8307ff326a8f64ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:29.220217 32460 ts_tablet_manager.cc:1875] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.221545 32458 ts_tablet_manager.cc:1888] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.221666 32458 log.cc:1196] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/1cff896decf14fca8307ff326a8f64ef
I20240502 14:00:29.222009 32458 ts_tablet_manager.cc:1909] T 1cff896decf14fca8307ff326a8f64ef P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:29.222426 32460 ts_tablet_manager.cc:1888] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.222506 32460 log.cc:1196] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/1cff896decf14fca8307ff326a8f64ef
I20240502 14:00:29.222762 32460 ts_tablet_manager.cc:1909] T 1cff896decf14fca8307ff326a8f64ef P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:29.223464 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 1cff896decf14fca8307ff326a8f64ef (table rebalancer_dynamic_table_new_name [id=506cb4e787674dfcac287afd5f15e457]) successfully deleted
I20240502 14:00:29.225030 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 1cff896decf14fca8307ff326a8f64ef (table rebalancer_dynamic_table_new_name [id=506cb4e787674dfcac287afd5f15e457]) successfully deleted
I20240502 14:00:29.227043 32254 raft_consensus.cc:1203] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.227396 753 tablet_replica.cc:330] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:29.230018 753 raft_consensus.cc:2229] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.230587 753 raft_consensus.cc:2258] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.231216 753 ts_tablet_manager.cc:1875] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.233878 753 ts_tablet_manager.cc:1888] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.233955 753 log.cc:1196] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/1cff896decf14fca8307ff326a8f64ef
I20240502 14:00:29.234305 753 ts_tablet_manager.cc:1909] T 1cff896decf14fca8307ff326a8f64ef P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:29.235885 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 1cff896decf14fca8307ff326a8f64ef (table rebalancer_dynamic_table_new_name [id=506cb4e787674dfcac287afd5f15e457]) successfully deleted
I20240502 14:00:29.238394 31991 raft_consensus.cc:1203] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.271741 32749 raft_consensus.cc:482] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.271868 32749 raft_consensus.cc:504] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.272308 32749 leader_election.cc:288] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:29.272711 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116d9f165b488796ab466f9a866471" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:29.272716 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116d9f165b488796ab466f9a866471" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:29.272843 31991 raft_consensus.cc:2454] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:29.272866 32254 raft_consensus.cc:2454] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:29.273144 32056 leader_election.cc:302] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.273388 32749 raft_consensus.cc:2781] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.273437 32749 raft_consensus.cc:482] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.273463 32749 raft_consensus.cc:3037] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.274276 32749 raft_consensus.cc:504] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.274641 32749 leader_election.cc:288] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:29.274978 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116d9f165b488796ab466f9a866471" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:29.274981 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116d9f165b488796ab466f9a866471" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:29.275077 32254 raft_consensus.cc:3037] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.275089 31991 raft_consensus.cc:3037] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.275797 32254 raft_consensus.cc:2454] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:29.275842 31991 raft_consensus.cc:2454] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:29.276057 32058 leader_election.cc:302] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.276245 32749 raft_consensus.cc:2781] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.276329 32749 raft_consensus.cc:686] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:29.276471 32749 consensus_queue.cc:237] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.278018 29328 catalog_manager.cc:5484] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:29.303042 29450 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.303997 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.305955 31972 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.306648 29842 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:29.308815 32254 raft_consensus.cc:1203] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.309201 31840 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.318917 32101 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:29.325328 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:29.326622 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.340536 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605]), partition=
I20240502 14:00:29.340536 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605]), partition=
I20240502 14:00:29.340806 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caed7e6b9ca4c9a96c280d28ea0af63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.340806 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caed7e6b9ca4c9a96c280d28ea0af63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.342914 692 tablet_bootstrap.cc:492] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:29.343254 29580 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:29.343448 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605]), partition=
I20240502 14:00:29.343672 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caed7e6b9ca4c9a96c280d28ea0af63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.343891 692 tablet_bootstrap.cc:654] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.345762 695 tablet_bootstrap.cc:492] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:29.346650 695 tablet_bootstrap.cc:654] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.361485 767 tablet_bootstrap.cc:492] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:29.362627 767 tablet_bootstrap.cc:654] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.365939 692 tablet_bootstrap.cc:492] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:29.366060 692 ts_tablet_manager.cc:1367] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:00:29.366583 692 raft_consensus.cc:348] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.366724 692 raft_consensus.cc:374] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.366765 692 raft_consensus.cc:729] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.366930 692 consensus_queue.cc:260] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.367218 692 ts_tablet_manager.cc:1398] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.371369 695 tablet_bootstrap.cc:492] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:29.371486 695 ts_tablet_manager.cc:1367] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20240502 14:00:29.371894 695 raft_consensus.cc:348] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.372015 695 raft_consensus.cc:374] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.372045 695 raft_consensus.cc:729] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.372184 695 consensus_queue.cc:260] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.372439 695 ts_tablet_manager.cc:1398] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.380906 767 tablet_bootstrap.cc:492] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:29.381150 767 ts_tablet_manager.cc:1367] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 14:00:29.381605 767 raft_consensus.cc:348] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.381707 767 raft_consensus.cc:374] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.381812 767 raft_consensus.cc:729] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.381968 767 consensus_queue.cc:260] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.382254 767 ts_tablet_manager.cc:1398] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.383946 32254 raft_consensus.cc:1203] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.412222 29732 consensus_queue.cc:237] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.413504 31860 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:29.413504 29863 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:29.413866 32631 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:29.414072 32626 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:29.415841 789 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:29.416209 29863 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:29.416184 31860 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:29.419615 29328 catalog_manager.cc:5484] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:29.422706 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 463dddd3473a42a68b488e1c465e997f with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:29.423012 32458 tablet_replica.cc:330] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:29.423115 32119 raft_consensus.cc:1203] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.423166 32458 raft_consensus.cc:2229] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.423341 32458 raft_consensus.cc:2258] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.423910 32458 ts_tablet_manager.cc:1875] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:29.426769 32458 ts_tablet_manager.cc:1888] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:29.426873 32458 log.cc:1196] T 463dddd3473a42a68b488e1c465e997f P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/463dddd3473a42a68b488e1c465e997f
I20240502 14:00:29.428500 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 463dddd3473a42a68b488e1c465e997f (table rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]) successfully deleted
I20240502 14:00:29.476343 29322 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 14:00:29.476521 29322 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 14:00:29.479794 29322 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet f9a7fdccb29d49499abaf50d0725153d
I20240502 14:00:29.480432 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a7fdccb29d49499abaf50d0725153d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:29.480582 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a7fdccb29d49499abaf50d0725153d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:29.480774 29322 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet 36daa6b40d8a48b8956b67a97833eff5
I20240502 14:00:29.480772 32458 tablet_replica.cc:330] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:29.480792 32457 tablet_replica.cc:330] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:29.480921 32457 raft_consensus.cc:2229] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.480949 32458 raft_consensus.cc:2229] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:29.481056 32457 raft_consensus.cc:2258] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.481166 32458 raft_consensus.cc:2258] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.481213 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 36daa6b40d8a48b8956b67a97833eff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:29.481271 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet 36daa6b40d8a48b8956b67a97833eff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:29.481467 29322 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet 7ebf61381a92430caa0c38cc04d14cb6
I20240502 14:00:29.481592 32458 ts_tablet_manager.cc:1875] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.481592 32457 ts_tablet_manager.cc:1875] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.481801 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ebf61381a92430caa0c38cc04d14cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:29.481792 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 36daa6b40d8a48b8956b67a97833eff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:29.482606 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ebf61381a92430caa0c38cc04d14cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:29.482790 673 tablet_replica.cc:330] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:29.482950 673 raft_consensus.cc:2229] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.483062 673 raft_consensus.cc:2258] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.483765 32458 ts_tablet_manager.cc:1888] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:29.483764 32457 ts_tablet_manager.cc:1888] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:29.483832 32457 log.cc:1196] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9a7fdccb29d49499abaf50d0725153d
I20240502 14:00:29.483832 32458 log.cc:1196] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9a7fdccb29d49499abaf50d0725153d
I20240502 14:00:29.484102 32457 ts_tablet_manager.cc:1909] T f9a7fdccb29d49499abaf50d0725153d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:29.484114 32458 ts_tablet_manager.cc:1909] T f9a7fdccb29d49499abaf50d0725153d P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:29.484823 32457 tablet_replica.cc:330] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:29.484961 32458 tablet_replica.cc:330] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:29.484987 32457 raft_consensus.cc:2229] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.485037 673 ts_tablet_manager.cc:1875] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.485095 32458 raft_consensus.cc:2229] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.485129 32457 raft_consensus.cc:2258] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.485226 32458 raft_consensus.cc:2258] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.485641 32457 ts_tablet_manager.cc:1875] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.485702 32458 ts_tablet_manager.cc:1875] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.486719 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet f9a7fdccb29d49499abaf50d0725153d (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.487114 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet f9a7fdccb29d49499abaf50d0725153d (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.487610 32457 ts_tablet_manager.cc:1888] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.487740 32457 log.cc:1196] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/36daa6b40d8a48b8956b67a97833eff5
I20240502 14:00:29.487892 32458 ts_tablet_manager.cc:1888] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.488003 32458 log.cc:1196] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/36daa6b40d8a48b8956b67a97833eff5
I20240502 14:00:29.488133 32457 ts_tablet_manager.cc:1909] T 36daa6b40d8a48b8956b67a97833eff5 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:29.488337 32458 ts_tablet_manager.cc:1909] T 36daa6b40d8a48b8956b67a97833eff5 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:29.488762 32457 tablet_replica.cc:330] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:29.488878 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 36daa6b40d8a48b8956b67a97833eff5 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.488965 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 36daa6b40d8a48b8956b67a97833eff5 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.488940 32457 raft_consensus.cc:2229] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.488751 673 ts_tablet_manager.cc:1888] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.489112 673 log.cc:1196] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/7ebf61381a92430caa0c38cc04d14cb6
I20240502 14:00:29.489212 32457 raft_consensus.cc:2258] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.489393 673 ts_tablet_manager.cc:1909] T 7ebf61381a92430caa0c38cc04d14cb6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:29.489697 32457 ts_tablet_manager.cc:1875] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.490025 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 7ebf61381a92430caa0c38cc04d14cb6 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.490833 32672 raft_consensus.cc:482] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.490958 32672 raft_consensus.cc:504] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.491429 32672 leader_election.cc:288] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:29.491719 32457 ts_tablet_manager.cc:1888] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.491851 32457 log.cc:1196] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/7ebf61381a92430caa0c38cc04d14cb6
I20240502 14:00:29.491904 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caed7e6b9ca4c9a96c280d28ea0af63" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:29.492055 31991 raft_consensus.cc:2454] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:29.492296 32457 ts_tablet_manager.cc:1909] T 7ebf61381a92430caa0c38cc04d14cb6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:29.492400 29797 leader_election.cc:302] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:29.492594 32672 raft_consensus.cc:2781] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.492643 32672 raft_consensus.cc:482] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.492678 32672 raft_consensus.cc:3037] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.493693 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 7ebf61381a92430caa0c38cc04d14cb6 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.493686 32672 raft_consensus.cc:504] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.494005 32672 leader_election.cc:288] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:29.494472 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caed7e6b9ca4c9a96c280d28ea0af63" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:29.494578 31991 raft_consensus.cc:3037] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.495357 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ebf61381a92430caa0c38cc04d14cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:29.495540 31991 raft_consensus.cc:2454] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:29.495981 32460 tablet_replica.cc:330] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:29.495997 29711 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ebf61381a92430caa0c38cc04d14cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:29.496134 32460 raft_consensus.cc:2229] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:29.496177 29797 leader_election.cc:302] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:29.496388 32460 raft_consensus.cc:2258] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.496886 32460 ts_tablet_manager.cc:1875] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.498975 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a7fdccb29d49499abaf50d0725153d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:29.495983 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet 36daa6b40d8a48b8956b67a97833eff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:29.501850 32672 raft_consensus.cc:2781] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.501931 32672 raft_consensus.cc:686] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:29.502043 833 tablet_replica.cc:330] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:29.502075 32672 consensus_queue.cc:237] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:29.502198 833 raft_consensus.cc:2229] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.503118 32460 ts_tablet_manager.cc:1888] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.503196 32460 log.cc:1196] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/7ebf61381a92430caa0c38cc04d14cb6
I20240502 14:00:29.504374 32460 ts_tablet_manager.cc:1909] T 7ebf61381a92430caa0c38cc04d14cb6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:29.504995 32460 tablet_replica.cc:330] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:29.505018 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 7ebf61381a92430caa0c38cc04d14cb6 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.506026 833 raft_consensus.cc:2258] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.506639 833 ts_tablet_manager.cc:1875] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.506769 32460 raft_consensus.cc:2229] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:29.507128 32460 raft_consensus.cc:2258] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.507668 32460 ts_tablet_manager.cc:1875] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.508503 29325 catalog_manager.cc:5484] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:29.509335 833 ts_tablet_manager.cc:1888] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.510313 32460 ts_tablet_manager.cc:1888] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.510403 32460 log.cc:1196] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/36daa6b40d8a48b8956b67a97833eff5
I20240502 14:00:29.510816 32460 ts_tablet_manager.cc:1909] T 36daa6b40d8a48b8956b67a97833eff5 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:29.511548 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 36daa6b40d8a48b8956b67a97833eff5 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.514436 833 log.cc:1196] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/7ebf61381a92430caa0c38cc04d14cb6
I20240502 14:00:29.514873 833 ts_tablet_manager.cc:1909] T 7ebf61381a92430caa0c38cc04d14cb6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:29.515651 833 tablet_replica.cc:330] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:29.515843 833 raft_consensus.cc:2229] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.516108 833 raft_consensus.cc:2258] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.516381 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 7ebf61381a92430caa0c38cc04d14cb6 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.516872 833 ts_tablet_manager.cc:1875] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.517863 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caed7e6b9ca4c9a96c280d28ea0af63" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:29.517943 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caed7e6b9ca4c9a96c280d28ea0af63" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:29.517995 31860 raft_consensus.cc:2454] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:29.519181 833 ts_tablet_manager.cc:1888] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:29.519717 831 tablet_replica.cc:330] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:29.519862 831 raft_consensus.cc:2229] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.519979 831 raft_consensus.cc:2258] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.520660 831 ts_tablet_manager.cc:1875] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:29.524725 831 ts_tablet_manager.cc:1888] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:29.528389 833 log.cc:1196] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9a7fdccb29d49499abaf50d0725153d
I20240502 14:00:29.528861 833 ts_tablet_manager.cc:1909] T f9a7fdccb29d49499abaf50d0725153d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:29.529671 831 log.cc:1196] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/36daa6b40d8a48b8956b67a97833eff5
I20240502 14:00:29.529942 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet f9a7fdccb29d49499abaf50d0725153d (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.530128 831 ts_tablet_manager.cc:1909] T 36daa6b40d8a48b8956b67a97833eff5 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:29.530658 29450 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.533380 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 36daa6b40d8a48b8956b67a97833eff5 (table rebalance_test_table_8 [id=4665c172d9114dfd9e83eeea42c0b5e5]) successfully deleted
I20240502 14:00:29.536829 29842 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:29.547011 31840 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.548522 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.549367 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:29.559628 32101 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:29.568830 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.570024 31972 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.606696 32749 consensus_queue.cc:1035] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.611246 32752 consensus_queue.cc:1035] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.613054 32749 consensus_queue.cc:1035] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.613754 32752 consensus_queue.cc:1035] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.628875 29863 consensus_queue.cc:237] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:29.630044 32254 raft_consensus.cc:1261] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:29.630079 29732 raft_consensus.cc:1261] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:29.630561 32672 consensus_queue.cc:1035] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:29.630859 31463 consensus_queue.cc:1035] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:29.632966 876 raft_consensus.cc:2932] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:29.633235 29732 raft_consensus.cc:2932] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:29.633263 32254 raft_consensus.cc:2932] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:29.636612 29325 catalog_manager.cc:5484] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:29.639362 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet e227b307a0f44a088bf50899d59f2118 with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:29.639757 32457 tablet_replica.cc:330] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:29.639920 32457 raft_consensus.cc:2229] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.640071 32457 raft_consensus.cc:2258] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.640571 32457 ts_tablet_manager.cc:1875] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:29.641527 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:29.642738 32457 ts_tablet_manager.cc:1888] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:29.642812 32457 log.cc:1196] T e227b307a0f44a088bf50899d59f2118 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/e227b307a0f44a088bf50899d59f2118
I20240502 14:00:29.643347 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet e227b307a0f44a088bf50899d59f2118 (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:29.646251 31861 raft_consensus.cc:3037] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.646250 31991 raft_consensus.cc:1261] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.647504 32672 consensus_queue.cc:1035] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.649773 29327 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (table rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605]). Expected version 1 got 0
I20240502 14:00:29.650537 891 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:29.650693 890 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:29.650938 891 tablet.cc:1620] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:29.651787 29327 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (table rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605]). Expected version 1 got 0
I20240502 14:00:29.652328 890 tablet.cc:1620] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:29.656373 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605] alter complete (version 1)
I20240502 14:00:29.667726 31861 raft_consensus.cc:1261] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.668226 32672 consensus_queue.cc:1035] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.669930 897 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:29.670346 897 tablet.cc:1620] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:29.671093 29325 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (table rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605]). Expected version 1 got 0
I20240502 14:00:29.688863 31861 raft_consensus.cc:1203] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:29.734644 29450 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.740617 29842 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:29.743304 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.748301 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:29.752911 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:29.756083 32101 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:29.763191 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:29.766348 31840 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.770138 31972 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:29.773903 890 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:29.774273 890 tablet.cc:1620] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:29.775250 897 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:29.775537 897 tablet.cc:1620] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:29.777231 32749 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.778936 891 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:29.779229 891 tablet.cc:1620] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:29.781188 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=4cf3adb40c4b4a6092a3b265eba97605] alter complete (version 2)
I20240502 14:00:29.792361 32752 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.793480 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:29.805105 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 1b37defc84d34af08f5744bf60a038c5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.805425 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b37defc84d34af08f5744bf60a038c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.806020 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 1b37defc84d34af08f5744bf60a038c5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.806193 32102 tablet_service.cc:1467] Processing CreateTablet for tablet fe523f0c364f447cb2a62b599d030cca (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.806289 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b37defc84d34af08f5744bf60a038c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.806430 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe523f0c364f447cb2a62b599d030cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.806020 32103 tablet_service.cc:1467] Processing CreateTablet for tablet ec8e4ac081604ec8a3f463f740d9119c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.806656 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8e4ac081604ec8a3f463f740d9119c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.808308 31841 tablet_service.cc:1467] Processing CreateTablet for tablet fe523f0c364f447cb2a62b599d030cca (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.809016 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe523f0c364f447cb2a62b599d030cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.811095 692 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:29.811444 31971 tablet_service.cc:1467] Processing CreateTablet for tablet fe523f0c364f447cb2a62b599d030cca (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.811715 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe523f0c364f447cb2a62b599d030cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.811954 692 tablet_bootstrap.cc:654] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.812175 31972 tablet_service.cc:1467] Processing CreateTablet for tablet ec8e4ac081604ec8a3f463f740d9119c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.812389 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8e4ac081604ec8a3f463f740d9119c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.814345 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 1b37defc84d34af08f5744bf60a038c5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.814560 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b37defc84d34af08f5744bf60a038c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.818029 939 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:29.818641 29450 tablet_service.cc:1467] Processing CreateTablet for tablet ec8e4ac081604ec8a3f463f740d9119c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.818977 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8e4ac081604ec8a3f463f740d9119c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.826040 938 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:29.826370 692 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:29.826470 692 ts_tablet_manager.cc:1367] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240502 14:00:29.826953 938 tablet_bootstrap.cc:654] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.826967 692 raft_consensus.cc:348] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.827097 692 raft_consensus.cc:374] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.827131 692 raft_consensus.cc:729] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.827251 692 consensus_queue.cc:260] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.827519 692 ts_tablet_manager.cc:1398] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.827653 692 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:29.828104 939 tablet_bootstrap.cc:654] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.828207 938 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:29.828287 938 ts_tablet_manager.cc:1367] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:29.828503 692 tablet_bootstrap.cc:654] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.828817 938 raft_consensus.cc:348] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.828923 938 raft_consensus.cc:374] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.829084 938 raft_consensus.cc:729] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.829228 938 consensus_queue.cc:260] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.829617 938 ts_tablet_manager.cc:1398] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.829823 938 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:29.830619 692 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:29.830713 692 ts_tablet_manager.cc:1367] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:29.831220 692 raft_consensus.cc:348] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.831323 692 raft_consensus.cc:374] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.831357 692 raft_consensus.cc:729] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.831465 692 consensus_queue.cc:260] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.831776 692 ts_tablet_manager.cc:1398] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.832113 939 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:29.832204 939 ts_tablet_manager.cc:1367] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:29.832551 938 tablet_bootstrap.cc:654] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.832837 939 raft_consensus.cc:348] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.832937 939 raft_consensus.cc:374] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.832971 939 raft_consensus.cc:729] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.833130 939 consensus_queue.cc:260] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.833925 939 ts_tablet_manager.cc:1398] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:29.834090 939 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:29.834975 695 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:29.835278 939 tablet_bootstrap.cc:654] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.835861 695 tablet_bootstrap.cc:654] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.836503 939 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:29.836580 939 ts_tablet_manager.cc:1367] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:29.836953 695 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:29.837028 695 ts_tablet_manager.cc:1367] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:29.837008 939 raft_consensus.cc:348] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.837105 939 raft_consensus.cc:374] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.837134 939 raft_consensus.cc:729] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.837240 939 consensus_queue.cc:260] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.837391 695 raft_consensus.cc:348] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.837489 695 raft_consensus.cc:374] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.837514 695 raft_consensus.cc:729] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.837520 939 ts_tablet_manager.cc:1398] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:29.837608 695 consensus_queue.cc:260] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.837680 939 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:29.837852 695 ts_tablet_manager.cc:1398] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:29.837976 695 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:29.838559 939 tablet_bootstrap.cc:654] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.838704 695 tablet_bootstrap.cc:654] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.839617 939 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:29.839709 939 ts_tablet_manager.cc:1367] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:29.840039 695 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:29.840103 695 ts_tablet_manager.cc:1367] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:29.840179 939 raft_consensus.cc:348] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.840276 939 raft_consensus.cc:374] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.840325 939 raft_consensus.cc:729] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.840451 939 consensus_queue.cc:260] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.840466 695 raft_consensus.cc:348] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.840545 695 raft_consensus.cc:374] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.840571 695 raft_consensus.cc:729] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.840687 695 consensus_queue.cc:260] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.840723 939 ts_tablet_manager.cc:1398] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:29.840915 695 ts_tablet_manager.cc:1398] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.846722 938 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:29.846824 938 ts_tablet_manager.cc:1367] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240502 14:00:29.847393 938 raft_consensus.cc:348] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.847517 938 raft_consensus.cc:374] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.847551 938 raft_consensus.cc:729] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.847689 938 consensus_queue.cc:260] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.848052 938 ts_tablet_manager.cc:1398] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:29.862600 937 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.862913 937 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.863400 937 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:29.863807 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:29.863967 31861 raft_consensus.cc:2454] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:29.864434 31925 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:29.864660 937 raft_consensus.cc:2781] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.864732 937 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.864790 937 raft_consensus.cc:3037] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.865658 937 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.866015 937 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:29.866338 29601 consensus_queue.cc:237] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.866387 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:29.866505 31861 raft_consensus.cc:3037] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.867560 31861 raft_consensus.cc:2454] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:29.867769 32119 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:29.867918 31925 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:29.868062 937 raft_consensus.cc:2781] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.868150 937 raft_consensus.cc:686] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:29.868269 937 consensus_queue.cc:237] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:29.868332 29471 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:29.868606 31131 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:29.868944 31131 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:29.870477 29327 catalog_manager.cc:5484] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:29.871888 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:29.872043 32119 raft_consensus.cc:2454] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:29.872149 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:29.872254 32122 raft_consensus.cc:3037] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.873180 32122 raft_consensus.cc:2454] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:29.873145 966 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:29.873808 29471 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:29.873853 32122 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:29.876380 29325 catalog_manager.cc:5484] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:29.882143 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 253ea421888249d180928ba0aef28ddd with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:29.883033 833 tablet_replica.cc:330] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:29.883227 833 raft_consensus.cc:2229] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:29.883394 833 raft_consensus.cc:2258] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:29.883991 833 ts_tablet_manager.cc:1875] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:29.886005 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:29.887245 833 ts_tablet_manager.cc:1888] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:29.887336 833 log.cc:1196] T 253ea421888249d180928ba0aef28ddd P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/253ea421888249d180928ba0aef28ddd
I20240502 14:00:29.888890 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 253ea421888249d180928ba0aef28ddd (table rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]) successfully deleted
I20240502 14:00:29.893011 890 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:29.893519 891 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:29.893571 897 tablet.cc:1722] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:29.899776 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=4cf3adb40c4b4a6092a3b265eba97605] alter complete (version 3)
I20240502 14:00:29.928964 937 raft_consensus.cc:482] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.929080 937 raft_consensus.cc:504] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.929517 937 leader_election.cc:288] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:29.929863 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8e4ac081604ec8a3f463f740d9119c" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:29.930025 32122 raft_consensus.cc:2454] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:29.930220 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8e4ac081604ec8a3f463f740d9119c" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:29.930370 29471 raft_consensus.cc:2454] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:29.930385 31925 leader_election.cc:302] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.930542 937 raft_consensus.cc:2781] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.930603 937 raft_consensus.cc:482] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.930641 937 raft_consensus.cc:3037] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.931277 31578 raft_consensus.cc:482] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.931370 31578 raft_consensus.cc:504] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.931780 31578 leader_election.cc:288] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:29.931751 937 raft_consensus.cc:504] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.932103 937 leader_election.cc:288] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:29.932667 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8e4ac081604ec8a3f463f740d9119c" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:29.932691 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8e4ac081604ec8a3f463f740d9119c" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:29.932683 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b37defc84d34af08f5744bf60a038c5" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:29.932778 29471 raft_consensus.cc:3037] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.932780 32122 raft_consensus.cc:3037] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.932823 31861 raft_consensus.cc:2454] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:29.933166 29404 leader_election.cc:302] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:29.933323 31578 raft_consensus.cc:2781] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.933373 31578 raft_consensus.cc:482] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.933424 31578 raft_consensus.cc:3037] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.933703 32122 raft_consensus.cc:2454] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:29.933703 29471 raft_consensus.cc:2454] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:29.933992 31925 leader_election.cc:302] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:29.934342 31578 raft_consensus.cc:504] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.934377 937 raft_consensus.cc:2781] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.934473 937 raft_consensus.cc:686] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:29.934659 31578 leader_election.cc:288] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:29.934616 937 consensus_queue.cc:237] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.935586 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b37defc84d34af08f5744bf60a038c5" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:29.935706 31861 raft_consensus.cc:3037] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.936690 31861 raft_consensus.cc:2454] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:29.937050 29404 leader_election.cc:302] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:29.937827 31578 raft_consensus.cc:2781] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.937916 31578 raft_consensus.cc:686] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:29.938048 31578 consensus_queue.cc:237] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:29.938123 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b37defc84d34af08f5744bf60a038c5" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:29.938232 32119 raft_consensus.cc:3037] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.939220 32119 raft_consensus.cc:2454] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:29.939556 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b37defc84d34af08f5744bf60a038c5" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:29.939682 32122 raft_consensus.cc:2362] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1. Re-sending same reply.
I20240502 14:00:29.940207 29325 catalog_manager.cc:5484] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:29.941339 29323 catalog_manager.cc:5484] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:29.993402 31972 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:29.995378 29450 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:29.995582 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:29.996270 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 14:00:29.996416 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 14:00:29.997306 29842 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:29.999423 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.001698 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet f4697431776e44a1907cf4b0865414da
I20240502 14:00:30.002024 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet f251ff0bea5c4f1f8f9f1e14ca8de60d
I20240502 14:00:30.002221 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet 6767168f6f2b4e15aab7e64b0eab57c3
I20240502 14:00:30.002740 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet f4697431776e44a1907cf4b0865414da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:30.002764 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:30.002912 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet 6767168f6f2b4e15aab7e64b0eab57c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:30.002928 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet f4697431776e44a1907cf4b0865414da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:30.003176 32457 tablet_replica.cc:330] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:30.003327 32457 raft_consensus.cc:2229] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.003407 1025 tablet_replica.cc:330] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:30.003479 32457 raft_consensus.cc:2258] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.003521 1025 raft_consensus.cc:2229] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.003609 1025 raft_consensus.cc:2258] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.003856 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:30.003960 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:30.004022 32457 ts_tablet_manager.cc:1875] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.004184 1025 ts_tablet_manager.cc:1875] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.006175 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet 6767168f6f2b4e15aab7e64b0eab57c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:30.006320 32457 ts_tablet_manager.cc:1888] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.006386 32457 log.cc:1196] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/f4697431776e44a1907cf4b0865414da
I20240502 14:00:30.006486 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:30.006656 32457 ts_tablet_manager.cc:1909] T f4697431776e44a1907cf4b0865414da P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:30.006780 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet f251ff0bea5c4f1f8f9f1e14ca8de60d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:30.006930 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet f4697431776e44a1907cf4b0865414da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:30.006980 833 tablet_replica.cc:330] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:30.007084 32460 tablet_replica.cc:330] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:30.007112 833 raft_consensus.cc:2229] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.007233 833 raft_consensus.cc:2258] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.007215 32460 raft_consensus.cc:2229] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.007369 32460 raft_consensus.cc:2258] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.007416 1025 ts_tablet_manager.cc:1888] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.007474 1025 log.cc:1196] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/f4697431776e44a1907cf4b0865414da
I20240502 14:00:30.007720 833 ts_tablet_manager.cc:1875] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.007736 1025 ts_tablet_manager.cc:1909] T f4697431776e44a1907cf4b0865414da P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:30.007921 32460 ts_tablet_manager.cc:1875] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.008404 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet f4697431776e44a1907cf4b0865414da (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.009780 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 6767168f6f2b4e15aab7e64b0eab57c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:30.009781 32460 ts_tablet_manager.cc:1888] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.009896 833 ts_tablet_manager.cc:1888] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.009941 32460 log.cc:1196] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/f251ff0bea5c4f1f8f9f1e14ca8de60d
I20240502 14:00:30.009970 833 log.cc:1196] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/f251ff0bea5c4f1f8f9f1e14ca8de60d
I20240502 14:00:30.010207 32460 ts_tablet_manager.cc:1909] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:30.010269 833 ts_tablet_manager.cc:1909] T f251ff0bea5c4f1f8f9f1e14ca8de60d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:30.010697 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 6caed7e6b9ca4c9a96c280d28ea0af63
I20240502 14:00:30.011001 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet f251ff0bea5c4f1f8f9f1e14ca8de60d (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.011171 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet f251ff0bea5c4f1f8f9f1e14ca8de60d (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.011298 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet f4697431776e44a1907cf4b0865414da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:30.011492 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 6767168f6f2b4e15aab7e64b0eab57c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:30.011870 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:30.011974 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:30.012027 831 tablet_replica.cc:330] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:30.012151 831 raft_consensus.cc:2229] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.012275 831 raft_consensus.cc:2258] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.012418 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caed7e6b9ca4c9a96c280d28ea0af63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:29 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:30.012957 831 ts_tablet_manager.cc:1875] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.014916 831 ts_tablet_manager.cc:1888] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.014978 831 log.cc:1196] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/6caed7e6b9ca4c9a96c280d28ea0af63
I20240502 14:00:30.015224 831 ts_tablet_manager.cc:1909] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:30.016500 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (table rebalancer_dynamic_table_new_name [id=4cf3adb40c4b4a6092a3b265eba97605]) successfully deleted
I20240502 14:00:30.010818 32460 tablet_replica.cc:330] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:30.017621 32460 raft_consensus.cc:2229] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.017978 32460 raft_consensus.cc:2258] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.020123 32460 ts_tablet_manager.cc:1875] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.022127 32460 ts_tablet_manager.cc:1888] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.022199 32460 log.cc:1196] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/f4697431776e44a1907cf4b0865414da
I20240502 14:00:30.025727 32460 ts_tablet_manager.cc:1909] T f4697431776e44a1907cf4b0865414da P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:30.026480 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet f4697431776e44a1907cf4b0865414da (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.026739 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:30.026980 32460 tablet_replica.cc:330] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:30.027170 32460 raft_consensus.cc:2229] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.027361 32460 raft_consensus.cc:2258] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.027474 1026 tablet_replica.cc:330] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:30.027648 1026 raft_consensus.cc:2229] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.027762 1026 raft_consensus.cc:2258] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.027930 32460 ts_tablet_manager.cc:1875] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.028237 1026 ts_tablet_manager.cc:1875] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.030040 1026 ts_tablet_manager.cc:1888] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.030109 1026 log.cc:1196] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/6767168f6f2b4e15aab7e64b0eab57c3
I20240502 14:00:30.030122 32460 ts_tablet_manager.cc:1888] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.030198 32460 log.cc:1196] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/6767168f6f2b4e15aab7e64b0eab57c3
I20240502 14:00:30.030449 1026 ts_tablet_manager.cc:1909] T 6767168f6f2b4e15aab7e64b0eab57c3 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:30.030480 32460 ts_tablet_manager.cc:1909] T 6767168f6f2b4e15aab7e64b0eab57c3 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:30.031078 32460 tablet_replica.cc:330] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:30.031193 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 6767168f6f2b4e15aab7e64b0eab57c3 (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.031253 32460 raft_consensus.cc:2229] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.031433 1026 tablet_replica.cc:330] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:30.031533 32460 raft_consensus.cc:2258] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.031579 1026 raft_consensus.cc:2229] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.031713 1026 raft_consensus.cc:2258] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.032246 1026 ts_tablet_manager.cc:1875] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.032117 32460 ts_tablet_manager.cc:1875] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.032254 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 6767168f6f2b4e15aab7e64b0eab57c3 (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.034016 1026 ts_tablet_manager.cc:1888] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.034132 1026 log.cc:1196] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/6caed7e6b9ca4c9a96c280d28ea0af63
I20240502 14:00:30.034320 32460 ts_tablet_manager.cc:1888] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.034413 32460 log.cc:1196] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/6caed7e6b9ca4c9a96c280d28ea0af63
I20240502 14:00:30.034543 1026 ts_tablet_manager.cc:1909] T 6caed7e6b9ca4c9a96c280d28ea0af63 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:30.034664 32460 ts_tablet_manager.cc:1909] T 6caed7e6b9ca4c9a96c280d28ea0af63 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:30.035156 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (table rebalancer_dynamic_table_new_name [id=4cf3adb40c4b4a6092a3b265eba97605]) successfully deleted
I20240502 14:00:30.037304 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 6caed7e6b9ca4c9a96c280d28ea0af63 (table rebalancer_dynamic_table_new_name [id=4cf3adb40c4b4a6092a3b265eba97605]) successfully deleted
I20240502 14:00:30.039247 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet f4697431776e44a1907cf4b0865414da (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.039470 32457 tablet_replica.cc:330] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:30.039664 32457 raft_consensus.cc:2229] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.039846 32457 raft_consensus.cc:2258] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.040357 32457 ts_tablet_manager.cc:1875] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.050132 32457 ts_tablet_manager.cc:1888] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.050550 32457 log.cc:1196] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/f251ff0bea5c4f1f8f9f1e14ca8de60d
I20240502 14:00:30.051586 32457 ts_tablet_manager.cc:1909] T f251ff0bea5c4f1f8f9f1e14ca8de60d P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:30.055099 32457 tablet_replica.cc:330] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:30.055197 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet f251ff0bea5c4f1f8f9f1e14ca8de60d (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.055826 32457 raft_consensus.cc:2229] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.056051 32457 raft_consensus.cc:2258] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.056535 32457 ts_tablet_manager.cc:1875] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.058537 32457 ts_tablet_manager.cc:1888] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.058604 32457 log.cc:1196] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/6767168f6f2b4e15aab7e64b0eab57c3
I20240502 14:00:30.058702 31841 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.058873 32457 ts_tablet_manager.cc:1909] T 6767168f6f2b4e15aab7e64b0eab57c3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:30.059305 32103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:30.059551 1027 tablet_replica.cc:330] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:30.059742 1027 raft_consensus.cc:2229] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.059928 1027 raft_consensus.cc:2258] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.060472 1027 ts_tablet_manager.cc:1875] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.063743 1027 ts_tablet_manager.cc:1888] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.063874 1027 log.cc:1196] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/f4697431776e44a1907cf4b0865414da
I20240502 14:00:30.064246 1027 ts_tablet_manager.cc:1909] T f4697431776e44a1907cf4b0865414da P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:30.064942 1027 tablet_replica.cc:330] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:30.064921 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet f4697431776e44a1907cf4b0865414da (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.065145 1027 raft_consensus.cc:2229] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.065337 1027 raft_consensus.cc:2258] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.065363 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 6767168f6f2b4e15aab7e64b0eab57c3 (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.065888 1027 ts_tablet_manager.cc:1875] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.067719 1027 ts_tablet_manager.cc:1888] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.067823 1027 log.cc:1196] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/6767168f6f2b4e15aab7e64b0eab57c3
I20240502 14:00:30.068166 1027 ts_tablet_manager.cc:1909] T 6767168f6f2b4e15aab7e64b0eab57c3 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:30.068768 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 6767168f6f2b4e15aab7e64b0eab57c3 (table rebalance_test_table_10 [id=04e8723873744e3e86a047ab11b5ea59]) successfully deleted
I20240502 14:00:30.112903 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:30.118474 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2410150ce34ce4b6ba1f7c619a13df (DEFAULT_TABLE table=rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8]), partition=
I20240502 14:00:30.118561 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2410150ce34ce4b6ba1f7c619a13df (DEFAULT_TABLE table=rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8]), partition=
I20240502 14:00:30.118760 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2410150ce34ce4b6ba1f7c619a13df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.118772 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2410150ce34ce4b6ba1f7c619a13df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.120944 695 tablet_bootstrap.cc:492] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:30.121429 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2410150ce34ce4b6ba1f7c619a13df (DEFAULT_TABLE table=rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8]), partition=
I20240502 14:00:30.121660 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2410150ce34ce4b6ba1f7c619a13df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.121783 695 tablet_bootstrap.cc:654] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.122294 692 tablet_bootstrap.cc:492] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:30.123483 692 tablet_bootstrap.cc:654] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.124485 692 tablet_bootstrap.cc:492] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:30.124569 692 ts_tablet_manager.cc:1367] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.124963 692 raft_consensus.cc:348] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.125065 692 raft_consensus.cc:374] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.125092 692 raft_consensus.cc:729] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.125200 692 consensus_queue.cc:260] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.125434 692 ts_tablet_manager.cc:1398] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.125946 695 tablet_bootstrap.cc:492] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:30.126030 695 ts_tablet_manager.cc:1367] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:30.126415 695 raft_consensus.cc:348] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.126503 695 raft_consensus.cc:374] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.126530 695 raft_consensus.cc:729] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.126643 695 consensus_queue.cc:260] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.126888 695 ts_tablet_manager.cc:1398] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:00:30.128672 1049 tablet_bootstrap.cc:492] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:30.129541 1049 tablet_bootstrap.cc:654] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.130813 1049 tablet_bootstrap.cc:492] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:30.130929 1049 ts_tablet_manager.cc:1367] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.131278 1049 raft_consensus.cc:348] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.131368 1049 raft_consensus.cc:374] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.131392 1049 raft_consensus.cc:729] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.131510 1049 consensus_queue.cc:260] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.131758 1049 ts_tablet_manager.cc:1398] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:30.292062 937 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.296110 937 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.352409 937 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.355665 937 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.395159 1075 raft_consensus.cc:482] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.395280 1075 raft_consensus.cc:504] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.395668 1075 leader_election.cc:288] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:30.395937 644 raft_consensus.cc:482] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.395967 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:30.396100 31991 raft_consensus.cc:2454] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:30.396057 644 raft_consensus.cc:504] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.396099 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:30.396238 32254 raft_consensus.cc:2454] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:30.396536 644 leader_election.cc:288] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:30.396688 31796 leader_election.cc:302] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:30.397094 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:30.397212 31861 raft_consensus.cc:2454] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:30.397686 32188 leader_election.cc:302] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:30.397836 644 raft_consensus.cc:2781] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.397894 644 raft_consensus.cc:482] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.397933 644 raft_consensus.cc:3037] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.397925 1075 raft_consensus.cc:2781] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.397974 1075 raft_consensus.cc:482] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.398011 1075 raft_consensus.cc:3037] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.398994 1075 raft_consensus.cc:504] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.399004 644 raft_consensus.cc:504] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.399340 1075 leader_election.cc:288] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:30.399331 644 leader_election.cc:288] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:30.399705 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:30.399865 31861 raft_consensus.cc:2379] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8973f2527acb48f5ad36589652de73e1 in current term 1: Already voted for candidate f0e01c4f581843e9ad4ced97f0072dc4 in this term.
I20240502 14:00:30.399863 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:30.399891 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:30.399962 31991 raft_consensus.cc:3037] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.400055 32254 raft_consensus.cc:2379] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f0e01c4f581843e9ad4ced97f0072dc4 in current term 1: Already voted for candidate 8973f2527acb48f5ad36589652de73e1 in this term.
I20240502 14:00:30.400831 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:30.400914 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2410150ce34ce4b6ba1f7c619a13df" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:30.400941 31991 raft_consensus.cc:2454] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:30.401293 32188 leader_election.cc:302] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:30.401453 644 raft_consensus.cc:2726] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:00:30.401546 31794 leader_election.cc:302] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters: 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:30.401690 1075 raft_consensus.cc:2781] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.401769 1075 raft_consensus.cc:686] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:30.401893 1075 consensus_queue.cc:237] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.403525 29325 catalog_manager.cc:5484] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:30.431888 31972 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:30.438879 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:30.442728 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:30.443181 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:30.451273 29325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:30.453640 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:30.454350 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.455297 31841 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:30.460697 32103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:30.462085 31972 tablet_service.cc:1467] Processing CreateTablet for tablet df1c6786a80e40a98130a5d9057e2877 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.462203 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9e9de44a046ffb4594755f44f15b8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.462386 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1c6786a80e40a98130a5d9057e2877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.462589 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9e9de44a046ffb4594755f44f15b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.462882 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 82cda992fe784d9f9015e6a3d431f1cd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.463141 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cda992fe784d9f9015e6a3d431f1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.464723 695 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:30.465425 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9e9de44a046ffb4594755f44f15b8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.465634 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9e9de44a046ffb4594755f44f15b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.465780 695 tablet_bootstrap.cc:654] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.467242 695 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:30.467312 695 ts_tablet_manager.cc:1367] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:30.467722 695 raft_consensus.cc:348] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.467818 695 raft_consensus.cc:374] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.467845 695 raft_consensus.cc:729] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.467950 695 consensus_queue.cc:260] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.468016 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 82cda992fe784d9f9015e6a3d431f1cd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.468199 695 ts_tablet_manager.cc:1398] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.468269 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cda992fe784d9f9015e6a3d431f1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.468538 32103 tablet_service.cc:1467] Processing CreateTablet for tablet df1c6786a80e40a98130a5d9057e2877 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.468758 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1c6786a80e40a98130a5d9057e2877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.468900 695 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:30.469808 695 tablet_bootstrap.cc:654] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.470938 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 82cda992fe784d9f9015e6a3d431f1cd (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.470995 1049 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:30.471144 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cda992fe784d9f9015e6a3d431f1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.471819 1049 tablet_bootstrap.cc:654] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.473763 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9e9de44a046ffb4594755f44f15b8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.474041 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9e9de44a046ffb4594755f44f15b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.475859 692 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:30.476768 692 tablet_bootstrap.cc:654] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.476838 31841 tablet_service.cc:1467] Processing CreateTablet for tablet df1c6786a80e40a98130a5d9057e2877 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.477043 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1c6786a80e40a98130a5d9057e2877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.483845 1049 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:30.483933 1049 ts_tablet_manager.cc:1367] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240502 14:00:30.484350 1049 raft_consensus.cc:348] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.484469 1049 raft_consensus.cc:374] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.484498 1049 raft_consensus.cc:729] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.484604 1049 consensus_queue.cc:260] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.484849 1049 ts_tablet_manager.cc:1398] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:30.485090 1097 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:30.487437 1097 tablet_bootstrap.cc:654] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.487474 692 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:30.487563 692 ts_tablet_manager.cc:1367] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:30.488041 692 raft_consensus.cc:348] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.488164 692 raft_consensus.cc:374] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.488196 692 raft_consensus.cc:729] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.488338 692 consensus_queue.cc:260] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.488612 692 ts_tablet_manager.cc:1398] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.490167 692 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:30.492792 692 tablet_bootstrap.cc:654] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.493870 692 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:30.493939 692 ts_tablet_manager.cc:1367] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:30.494452 692 raft_consensus.cc:348] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.494561 692 raft_consensus.cc:374] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.494589 692 raft_consensus.cc:729] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.494688 692 consensus_queue.cc:260] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.497167 835 consensus_queue.cc:1035] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.498267 692 ts_tablet_manager.cc:1398] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:30.504376 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 14:00:30.504513 1094 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:30.504627 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 14:00:30.504778 695 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:30.504879 695 ts_tablet_manager.cc:1367] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240502 14:00:30.505297 695 raft_consensus.cc:348] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.505411 695 raft_consensus.cc:374] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.505439 695 raft_consensus.cc:729] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.505556 1094 tablet_bootstrap.cc:654] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.505554 695 consensus_queue.cc:260] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.505789 695 ts_tablet_manager.cc:1398] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:30.507815 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 4 replicas of tablet c35dda6845eb484886eed36e40b9231c
I20240502 14:00:30.508240 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet bcd9a7f130364a46b4a48f96bb28f7a6
I20240502 14:00:30.508468 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet c35dda6845eb484886eed36e40b9231c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:30.508596 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet add1fc32951945488dfb045e56dcecc9
I20240502 14:00:30.508688 32460 tablet_replica.cc:330] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:30.508693 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet c35dda6845eb484886eed36e40b9231c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:30.508710 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet c35dda6845eb484886eed36e40b9231c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:30.508829 32460 raft_consensus.cc:2229] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.508863 1026 tablet_replica.cc:330] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:30.508997 1026 raft_consensus.cc:2229] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.509014 32460 raft_consensus.cc:2258] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.509106 1026 raft_consensus.cc:2258] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.509208 32457 tablet_replica.cc:330] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:30.509347 32457 raft_consensus.cc:2229] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.509496 32457 raft_consensus.cc:2258] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.509503 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet add1fc32951945488dfb045e56dcecc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:30.509582 32460 ts_tablet_manager.cc:1875] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.509786 1026 ts_tablet_manager.cc:1875] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.509935 29711 tablet_service.cc:1514] Processing DeleteTablet for tablet add1fc32951945488dfb045e56dcecc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:30.509996 32457 ts_tablet_manager.cc:1875] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.510103 833 tablet_replica.cc:330] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:30.510232 833 raft_consensus.cc:2229] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.510428 833 raft_consensus.cc:2258] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.510972 833 ts_tablet_manager.cc:1875] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.511591 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd9a7f130364a46b4a48f96bb28f7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:30.512182 32460 ts_tablet_manager.cc:1888] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.512270 32460 log.cc:1196] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/c35dda6845eb484886eed36e40b9231c
I20240502 14:00:30.512543 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd9a7f130364a46b4a48f96bb28f7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:30.512598 32457 ts_tablet_manager.cc:1888] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.512620 1026 ts_tablet_manager.cc:1888] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.512655 32457 log.cc:1196] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/c35dda6845eb484886eed36e40b9231c
I20240502 14:00:30.512683 1026 log.cc:1196] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/c35dda6845eb484886eed36e40b9231c
I20240502 14:00:30.512905 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet c35dda6845eb484886eed36e40b9231c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:30.512974 32457 ts_tablet_manager.cc:1909] T c35dda6845eb484886eed36e40b9231c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:30.513193 833 ts_tablet_manager.cc:1888] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.513259 1026 ts_tablet_manager.cc:1909] T c35dda6845eb484886eed36e40b9231c P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:30.513268 833 log.cc:1196] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/add1fc32951945488dfb045e56dcecc9
I20240502 14:00:30.513520 833 ts_tablet_manager.cc:1909] T add1fc32951945488dfb045e56dcecc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:30.514024 833 tablet_replica.cc:330] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:30.514132 833 raft_consensus.cc:2229] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.514235 833 raft_consensus.cc:2258] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.514233 1027 tablet_replica.cc:330] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:30.514361 1027 raft_consensus.cc:2229] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.514537 1027 raft_consensus.cc:2258] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.514639 833 ts_tablet_manager.cc:1875] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.514997 1027 ts_tablet_manager.cc:1875] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.515216 1097 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:30.515314 1097 ts_tablet_manager.cc:1367] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 14:00:30.515797 1097 raft_consensus.cc:348] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.515843 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet add1fc32951945488dfb045e56dcecc9 (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.515923 1097 raft_consensus.cc:374] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.515959 1097 raft_consensus.cc:729] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.516099 1097 consensus_queue.cc:260] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.516407 1097 ts_tablet_manager.cc:1398] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:30.516669 1097 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:30.517089 1027 ts_tablet_manager.cc:1888] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.517155 1027 log.cc:1196] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/bcd9a7f130364a46b4a48f96bb28f7a6
I20240502 14:00:30.517573 1097 tablet_bootstrap.cc:654] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.517714 1027 ts_tablet_manager.cc:1909] T bcd9a7f130364a46b4a48f96bb28f7a6 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:30.518244 1027 tablet_replica.cc:330] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:30.518954 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet add1fc32951945488dfb045e56dcecc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:30.519034 1027 raft_consensus.cc:2229] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.519233 1027 raft_consensus.cc:2258] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.519367 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd9a7f130364a46b4a48f96bb28f7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:30.520537 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet bcd9a7f130364a46b4a48f96bb28f7a6 (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.520685 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet c35dda6845eb484886eed36e40b9231c (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.523001 1027 ts_tablet_manager.cc:1875] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.524423 1097 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:30.524554 1097 ts_tablet_manager.cc:1367] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:30.524724 835 consensus_queue.cc:1035] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.524977 1097 raft_consensus.cc:348] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.525075 1097 raft_consensus.cc:374] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.525103 1097 raft_consensus.cc:729] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.525203 1097 consensus_queue.cc:260] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.525732 32457 tablet_replica.cc:330] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:30.525868 32457 raft_consensus.cc:2229] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.525993 32457 raft_consensus.cc:2258] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.526471 32457 ts_tablet_manager.cc:1875] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.527299 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet c35dda6845eb484886eed36e40b9231c (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.527726 32460 ts_tablet_manager.cc:1909] T c35dda6845eb484886eed36e40b9231c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:30.527101 1097 ts_tablet_manager.cc:1398] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.528431 32460 tablet_replica.cc:330] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:30.528617 32460 raft_consensus.cc:2229] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.528697 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet c35dda6845eb484886eed36e40b9231c (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.528762 32460 raft_consensus.cc:2258] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.529290 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:30.530762 32457 ts_tablet_manager.cc:1888] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.531234 32460 ts_tablet_manager.cc:1875] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.531598 833 ts_tablet_manager.cc:1888] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.531668 833 log.cc:1196] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/bcd9a7f130364a46b4a48f96bb28f7a6
I20240502 14:00:30.533582 32460 ts_tablet_manager.cc:1888] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.533656 32460 log.cc:1196] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/add1fc32951945488dfb045e56dcecc9
I20240502 14:00:30.533798 32254 raft_consensus.cc:1261] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.533926 32460 ts_tablet_manager.cc:1909] T add1fc32951945488dfb045e56dcecc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:30.533982 833 ts_tablet_manager.cc:1909] T bcd9a7f130364a46b4a48f96bb28f7a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:30.534322 31990 raft_consensus.cc:1261] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.534565 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet add1fc32951945488dfb045e56dcecc9 (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.534929 1075 consensus_queue.cc:1035] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.535280 1027 ts_tablet_manager.cc:1888] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.535362 1027 log.cc:1196] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/c35dda6845eb484886eed36e40b9231c
I20240502 14:00:30.530823 32457 log.cc:1196] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/add1fc32951945488dfb045e56dcecc9
I20240502 14:00:30.535879 32457 ts_tablet_manager.cc:1909] T add1fc32951945488dfb045e56dcecc9 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:30.536442 32457 tablet_replica.cc:330] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:30.536562 32457 raft_consensus.cc:2229] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.536672 32457 raft_consensus.cc:2258] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.536924 1075 consensus_queue.cc:1035] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.537130 32457 ts_tablet_manager.cc:1875] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.537600 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet add1fc32951945488dfb045e56dcecc9 (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.538146 1027 ts_tablet_manager.cc:1909] T c35dda6845eb484886eed36e40b9231c P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:30.538241 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet bcd9a7f130364a46b4a48f96bb28f7a6 (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.538879 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet c35dda6845eb484886eed36e40b9231c (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.540104 32457 ts_tablet_manager.cc:1888] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:30.540203 32457 log.cc:1196] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/bcd9a7f130364a46b4a48f96bb28f7a6
I20240502 14:00:30.540653 32457 ts_tablet_manager.cc:1909] T bcd9a7f130364a46b4a48f96bb28f7a6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:30.541606 29329 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 2c2410150ce34ce4b6ba1f7c619a13df (table rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8]). Expected version 1 got 0
I20240502 14:00:30.542243 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet bcd9a7f130364a46b4a48f96bb28f7a6 (table rebalance_test_table_12 [id=522ca29703814f96b327672a3d1bffa6]) successfully deleted
I20240502 14:00:30.543203 29329 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 2c2410150ce34ce4b6ba1f7c619a13df (table rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8]). Expected version 1 got 0
I20240502 14:00:30.544134 855 mvcc.cc:205] Tried to move back new op lower bound from 7023240931462602752 to 7023240930927230976. Current Snapshot: MvccSnapshot[applied={T|T < 7023240931462602752}]
I20240502 14:00:30.546448 29320 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 2c2410150ce34ce4b6ba1f7c619a13df (table rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8]). Expected version 1 got 0
I20240502 14:00:30.546396 1110 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:30.546736 1110 tablet.cc:1620] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:30.548259 29320 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 2c2410150ce34ce4b6ba1f7c619a13df (table rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8]). Expected version 1 got 0
I20240502 14:00:30.550977 1116 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:30.551319 1116 tablet.cc:1620] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:30.551702 1111 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:30.552126 1111 tablet.cc:1620] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:30.552686 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8] alter complete (version 1)
I20240502 14:00:30.555271 32752 raft_consensus.cc:482] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.555361 32752 raft_consensus.cc:504] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.555756 32752 leader_election.cc:288] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:30.556250 1094 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:30.556365 1094 ts_tablet_manager.cc:1367] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.000s
I20240502 14:00:30.556990 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cda992fe784d9f9015e6a3d431f1cd" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:30.557116 32254 raft_consensus.cc:2454] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:30.557137 1094 raft_consensus.cc:348] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.557236 1094 raft_consensus.cc:374] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.557269 1094 raft_consensus.cc:729] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.557397 1094 consensus_queue.cc:260] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.557694 1094 ts_tablet_manager.cc:1398] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.557869 1094 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:30.558795 1094 tablet_bootstrap.cc:654] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.560712 32058 leader_election.cc:302] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:30.560925 32752 raft_consensus.cc:2781] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.560971 32752 raft_consensus.cc:482] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.561003 32752 raft_consensus.cc:3037] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.561898 32752 raft_consensus.cc:504] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.562235 32752 leader_election.cc:288] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:30.562534 1094 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:30.562620 1094 ts_tablet_manager.cc:1367] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:30.562707 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cda992fe784d9f9015e6a3d431f1cd" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:30.562805 32254 raft_consensus.cc:3037] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.563163 1094 raft_consensus.cc:348] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.563264 1094 raft_consensus.cc:374] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.563294 1094 raft_consensus.cc:729] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.563406 1094 consensus_queue.cc:260] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.563689 32254 raft_consensus.cc:2454] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:30.563730 1094 ts_tablet_manager.cc:1398] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.565806 32058 leader_election.cc:302] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:30.567605 32752 raft_consensus.cc:2781] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.567688 32752 raft_consensus.cc:686] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:30.567799 32752 consensus_queue.cc:237] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.569677 29325 catalog_manager.cc:5484] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:30.570500 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cda992fe784d9f9015e6a3d431f1cd" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:30.570606 29470 raft_consensus.cc:3037] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.571646 29470 raft_consensus.cc:2454] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:30.571815 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cda992fe784d9f9015e6a3d431f1cd" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:30.571923 29471 raft_consensus.cc:2362] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1. Re-sending same reply.
I20240502 14:00:30.581120 29601 consensus_queue.cc:237] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:30.582265 32122 raft_consensus.cc:1261] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:30.582372 29863 raft_consensus.cc:1261] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:30.582762 31131 consensus_queue.cc:1035] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:30.582965 1132 consensus_queue.cc:1035] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:30.584755 1135 raft_consensus.cc:2932] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:30.585179 29863 raft_consensus.cc:2932] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:30.585198 32122 raft_consensus.cc:2932] T 0867c6a22c2c4efd9a9babff371cc8ba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:30.588358 29325 catalog_manager.cc:5484] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:30.591110 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 0867c6a22c2c4efd9a9babff371cc8ba with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:30.591408 833 tablet_replica.cc:330] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:30.591488 858 raft_consensus.cc:482] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.591547 833 raft_consensus.cc:2229] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.591595 858 raft_consensus.cc:504] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.591679 833 raft_consensus.cc:2258] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.591995 858 leader_election.cc:288] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:30.592168 833 ts_tablet_manager.cc:1875] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:30.594151 833 ts_tablet_manager.cc:1888] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:30.594215 833 log.cc:1196] T 0867c6a22c2c4efd9a9babff371cc8ba P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/0867c6a22c2c4efd9a9babff371cc8ba
I20240502 14:00:30.594676 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c6786a80e40a98130a5d9057e2877" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:30.594738 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 0867c6a22c2c4efd9a9babff371cc8ba (table rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]) successfully deleted
I20240502 14:00:30.594808 31990 raft_consensus.cc:2454] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:30.595263 31794 leader_election.cc:302] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:30.595459 858 raft_consensus.cc:2781] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.595527 858 raft_consensus.cc:482] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.595573 858 raft_consensus.cc:3037] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.596576 858 raft_consensus.cc:504] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.596596 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c6786a80e40a98130a5d9057e2877" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:30.596693 32122 raft_consensus.cc:2454] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:30.596930 858 leader_election.cc:288] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:30.597365 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c6786a80e40a98130a5d9057e2877" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:30.597484 31990 raft_consensus.cc:3037] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.597625 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c6786a80e40a98130a5d9057e2877" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:30.597729 32122 raft_consensus.cc:3037] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.598431 31990 raft_consensus.cc:2454] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:30.598649 32122 raft_consensus.cc:2454] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:30.598691 31794 leader_election.cc:302] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:30.598891 858 raft_consensus.cc:2781] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.598997 858 raft_consensus.cc:686] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:30.599149 858 consensus_queue.cc:237] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:30.600982 29320 catalog_manager.cc:5484] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:30.658666 29324 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:30.664991 1110 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:30.665339 1110 tablet.cc:1620] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:30.665315 1116 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:30.665607 1116 tablet.cc:1620] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:30.667322 1111 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:30.667613 1111 tablet.cc:1620] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:30.672888 29324 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=392059e00b224eea94b12ca48d4f35b8] alter complete (version 2)
I20240502 14:00:30.754551 1105 raft_consensus.cc:482] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.754691 1105 raft_consensus.cc:504] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.755209 1105 leader_election.cc:288] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:30.755956 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9e9de44a046ffb4594755f44f15b8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:30.756103 31861 raft_consensus.cc:2454] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:30.756412 29404 leader_election.cc:302] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:30.756587 1105 raft_consensus.cc:2781] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.756644 1105 raft_consensus.cc:482] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.756695 1105 raft_consensus.cc:3037] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.757761 1105 raft_consensus.cc:504] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.758015 1105 leader_election.cc:288] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:30.758320 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9e9de44a046ffb4594755f44f15b8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:30.758438 31861 raft_consensus.cc:3037] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.759490 31861 raft_consensus.cc:2454] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:30.759716 29404 leader_election.cc:302] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:30.759881 1105 raft_consensus.cc:2781] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.759964 1105 raft_consensus.cc:686] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:30.760000 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9e9de44a046ffb4594755f44f15b8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:30.760078 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9e9de44a046ffb4594755f44f15b8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:30.760115 1105 consensus_queue.cc:237] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:30.760152 31990 raft_consensus.cc:2454] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:30.761907 29322 catalog_manager.cc:5484] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:30.775811 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:30.784358 1110 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:30.784945 1116 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:30.785372 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:30.785606 1111 tablet.cc:1722] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:30.789734 29449 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:30.792914 29324 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=392059e00b224eea94b12ca48d4f35b8] alter complete (version 3)
I20240502 14:00:30.795289 31972 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:30.800026 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.802098 32101 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:30.809052 31841 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:30.813323 29582 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:30.814069 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:30.866220 29471 consensus_queue.cc:237] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:30.867642 32255 raft_consensus.cc:1261] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:30.867727 29601 raft_consensus.cc:1261] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:30.868062 835 consensus_queue.cc:1035] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:30.868257 1105 consensus_queue.cc:1035] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:30.869968 835 raft_consensus.cc:2932] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:30.870362 32255 raft_consensus.cc:2932] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:30.871054 29601 raft_consensus.cc:2932] T b2c754e60c1d4f6f95a807e9644bd21f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:30.872921 29322 catalog_manager.cc:5484] T b2c754e60c1d4f6f95a807e9644bd21f P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:30.875793 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c754e60c1d4f6f95a807e9644bd21f with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:30.876029 833 tablet_replica.cc:330] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:30.876181 833 raft_consensus.cc:2229] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.876319 833 raft_consensus.cc:2258] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.876859 833 ts_tablet_manager.cc:1875] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:30.878962 833 ts_tablet_manager.cc:1888] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:30.879042 833 log.cc:1196] T b2c754e60c1d4f6f95a807e9644bd21f P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/b2c754e60c1d4f6f95a807e9644bd21f
I20240502 14:00:30.883263 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet b2c754e60c1d4f6f95a807e9644bd21f (table rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]) successfully deleted
I20240502 14:00:30.894872 29324 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:30.895059 29324 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:30.897292 29324 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 2c2410150ce34ce4b6ba1f7c619a13df
I20240502 14:00:30.898012 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2410150ce34ce4b6ba1f7c619a13df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:30.898010 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2410150ce34ce4b6ba1f7c619a13df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:30.898231 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2410150ce34ce4b6ba1f7c619a13df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:30 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:30.898523 1026 tablet_replica.cc:330] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:30.898677 1026 raft_consensus.cc:2229] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:30.898717 1225 tablet_replica.cc:330] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:30.898833 1225 raft_consensus.cc:2229] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.899010 1225 raft_consensus.cc:2258] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.899052 1026 raft_consensus.cc:2258] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.899663 1225 ts_tablet_manager.cc:1875] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.899685 1026 ts_tablet_manager.cc:1875] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.900925 1224 tablet_replica.cc:330] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:30.901136 1224 raft_consensus.cc:2229] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:30.901314 1224 raft_consensus.cc:2258] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:30.901744 1026 ts_tablet_manager.cc:1888] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.901867 1026 log.cc:1196] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/2c2410150ce34ce4b6ba1f7c619a13df
I20240502 14:00:30.901981 1224 ts_tablet_manager.cc:1875] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:30.901988 1225 ts_tablet_manager.cc:1888] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.902056 1225 log.cc:1196] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/2c2410150ce34ce4b6ba1f7c619a13df
I20240502 14:00:30.902184 1026 ts_tablet_manager.cc:1909] T 2c2410150ce34ce4b6ba1f7c619a13df P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:30.902400 1225 ts_tablet_manager.cc:1909] T 2c2410150ce34ce4b6ba1f7c619a13df P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:30.902927 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 2c2410150ce34ce4b6ba1f7c619a13df (table rebalancer_dynamic_table_new_name [id=392059e00b224eea94b12ca48d4f35b8]) successfully deleted
I20240502 14:00:30.903086 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 2c2410150ce34ce4b6ba1f7c619a13df (table rebalancer_dynamic_table_new_name [id=392059e00b224eea94b12ca48d4f35b8]) successfully deleted
I20240502 14:00:30.904129 1224 ts_tablet_manager.cc:1888] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:30.904196 1224 log.cc:1196] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/2c2410150ce34ce4b6ba1f7c619a13df
I20240502 14:00:30.904414 1224 ts_tablet_manager.cc:1909] T 2c2410150ce34ce4b6ba1f7c619a13df P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:30.905053 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 2c2410150ce34ce4b6ba1f7c619a13df (table rebalancer_dynamic_table_new_name [id=392059e00b224eea94b12ca48d4f35b8]) successfully deleted
I20240502 14:00:30.964479 31972 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:30.967728 31841 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:30.968840 32235 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.971856 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:30.976318 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:30.978381 29449 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:30.993458 32101 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:30.993610 1179 consensus_queue.cc:1035] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.993716 29582 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:30.999362 1075 consensus_queue.cc:1035] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.002794 29325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:31.009765 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 14:00:31.009915 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 14:00:31.011754 29713 tablet_service.cc:1467] Processing CreateTablet for tablet bd5d56ea730b4f08886cb6298e9786fa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45]), partition=
I20240502 14:00:31.012066 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5d56ea730b4f08886cb6298e9786fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.012061 31841 tablet_service.cc:1467] Processing CreateTablet for tablet bd5d56ea730b4f08886cb6298e9786fa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45]), partition=
I20240502 14:00:31.012346 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5d56ea730b4f08886cb6298e9786fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.013025 32235 tablet_service.cc:1467] Processing CreateTablet for tablet bd5d56ea730b4f08886cb6298e9786fa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45]), partition=
I20240502 14:00:31.013262 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5d56ea730b4f08886cb6298e9786fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.020327 1268 tablet_bootstrap.cc:492] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:31.021350 1268 tablet_bootstrap.cc:654] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.022085 1269 tablet_bootstrap.cc:492] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:31.023519 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet b1323869da8c4249b727852cb1a454b3
I20240502 14:00:31.023798 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 7808510875f441fd9814121434f534e0
I20240502 14:00:31.023963 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 2fed302438544dcf9c7e90b6a98576b3
I20240502 14:00:31.024441 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet b1323869da8c4249b727852cb1a454b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:31.024492 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fed302438544dcf9c7e90b6a98576b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:31.024659 1027 tablet_replica.cc:330] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:31.024643 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet b1323869da8c4249b727852cb1a454b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:31.024804 1027 raft_consensus.cc:2229] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.024946 1027 raft_consensus.cc:2258] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.025038 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 7808510875f441fd9814121434f534e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:31.025171 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fed302438544dcf9c7e90b6a98576b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:31.025494 1027 ts_tablet_manager.cc:1875] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.025605 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 7808510875f441fd9814121434f534e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:31.025969 29711 tablet_service.cc:1514] Processing DeleteTablet for tablet 7808510875f441fd9814121434f534e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:31.026190 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet b1323869da8c4249b727852cb1a454b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:31.026331 833 tablet_replica.cc:330] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:31.026461 833 raft_consensus.cc:2229] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.026580 833 raft_consensus.cc:2258] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.026938 32457 tablet_replica.cc:330] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:31.027076 32457 raft_consensus.cc:2229] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:31.027282 32457 raft_consensus.cc:2258] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.027428 833 ts_tablet_manager.cc:1875] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.027761 32457 ts_tablet_manager.cc:1875] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.029755 833 ts_tablet_manager.cc:1888] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.029826 833 log.cc:1196] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/7808510875f441fd9814121434f534e0
I20240502 14:00:31.029891 32457 ts_tablet_manager.cc:1888] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.029955 32457 log.cc:1196] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/b1323869da8c4249b727852cb1a454b3
I20240502 14:00:31.030146 833 ts_tablet_manager.cc:1909] T 7808510875f441fd9814121434f534e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:31.030201 32457 ts_tablet_manager.cc:1909] T b1323869da8c4249b727852cb1a454b3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:31.030653 32460 tablet_replica.cc:330] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:31.030738 833 tablet_replica.cc:330] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:31.030802 32460 raft_consensus.cc:2229] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.030947 32460 raft_consensus.cc:2258] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.031391 32460 ts_tablet_manager.cc:1875] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.033537 32460 ts_tablet_manager.cc:1888] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.033605 32460 log.cc:1196] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/7808510875f441fd9814121434f534e0
I20240502 14:00:31.033612 1267 tablet_bootstrap.cc:492] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:31.033793 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 7808510875f441fd9814121434f534e0 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.033861 32460 ts_tablet_manager.cc:1909] T 7808510875f441fd9814121434f534e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:31.033951 833 raft_consensus.cc:2229] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.034101 833 raft_consensus.cc:2258] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.034468 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fed302438544dcf9c7e90b6a98576b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:31.034515 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 7808510875f441fd9814121434f534e0 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.034914 833 ts_tablet_manager.cc:1875] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.035933 1027 ts_tablet_manager.cc:1888] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.035998 1027 log.cc:1196] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/b1323869da8c4249b727852cb1a454b3
I20240502 14:00:31.036254 1027 ts_tablet_manager.cc:1909] T b1323869da8c4249b727852cb1a454b3 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:31.036814 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet b1323869da8c4249b727852cb1a454b3 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.036922 833 ts_tablet_manager.cc:1888] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.036994 833 log.cc:1196] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/b1323869da8c4249b727852cb1a454b3
I20240502 14:00:31.037098 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet b1323869da8c4249b727852cb1a454b3 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.037130 1027 tablet_replica.cc:330] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:31.037300 1027 raft_consensus.cc:2229] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:31.037410 32457 tablet_replica.cc:330] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:31.037597 32457 raft_consensus.cc:2229] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:31.037638 1027 raft_consensus.cc:2258] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.037844 32457 raft_consensus.cc:2258] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.038142 1027 ts_tablet_manager.cc:1875] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.038414 32457 ts_tablet_manager.cc:1875] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.039347 833 ts_tablet_manager.cc:1909] T b1323869da8c4249b727852cb1a454b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:31.040026 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet b1323869da8c4249b727852cb1a454b3 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.040089 1027 ts_tablet_manager.cc:1888] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.040202 1027 log.cc:1196] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/2fed302438544dcf9c7e90b6a98576b3
I20240502 14:00:31.040239 833 tablet_replica.cc:330] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:31.040390 833 raft_consensus.cc:2229] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.040463 1267 tablet_bootstrap.cc:654] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.040501 1027 ts_tablet_manager.cc:1909] T 2fed302438544dcf9c7e90b6a98576b3 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:31.040529 833 raft_consensus.cc:2258] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.040968 833 ts_tablet_manager.cc:1875] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.041129 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 2fed302438544dcf9c7e90b6a98576b3 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.042750 833 ts_tablet_manager.cc:1888] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.043098 1269 tablet_bootstrap.cc:654] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.043917 833 log.cc:1196] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/2fed302438544dcf9c7e90b6a98576b3
I20240502 14:00:31.043468 1268 tablet_bootstrap.cc:492] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:31.044391 1268 ts_tablet_manager.cc:1367] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:00:31.045032 1268 raft_consensus.cc:348] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.045158 1268 raft_consensus.cc:374] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.045187 1268 raft_consensus.cc:729] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.045308 1268 consensus_queue.cc:260] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.045603 1268 ts_tablet_manager.cc:1398] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.046080 1269 tablet_bootstrap.cc:492] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:31.046173 1269 ts_tablet_manager.cc:1367] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:00:31.046588 1269 raft_consensus.cc:348] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.046989 833 ts_tablet_manager.cc:1909] T 2fed302438544dcf9c7e90b6a98576b3 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:31.047602 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 2fed302438544dcf9c7e90b6a98576b3 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.047748 1269 raft_consensus.cc:374] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.047842 1269 raft_consensus.cc:729] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.048017 1269 consensus_queue.cc:260] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.048370 1269 ts_tablet_manager.cc:1398] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:31.049778 32457 ts_tablet_manager.cc:1888] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.051221 32457 log.cc:1196] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/7808510875f441fd9814121434f534e0
I20240502 14:00:31.051550 32457 ts_tablet_manager.cc:1909] T 7808510875f441fd9814121434f534e0 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:31.052174 32457 tablet_replica.cc:330] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:31.052305 32457 raft_consensus.cc:2229] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.052433 32457 raft_consensus.cc:2258] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.052624 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 7808510875f441fd9814121434f534e0 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.052912 32457 ts_tablet_manager.cc:1875] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.054502 32457 ts_tablet_manager.cc:1888] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.054581 32457 log.cc:1196] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/2fed302438544dcf9c7e90b6a98576b3
I20240502 14:00:31.054811 32457 ts_tablet_manager.cc:1909] T 2fed302438544dcf9c7e90b6a98576b3 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:31.055562 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 2fed302438544dcf9c7e90b6a98576b3 (table rebalance_test_table_14 [id=d96b2730da534b308ada14baacdf879f]) successfully deleted
I20240502 14:00:31.062217 1267 tablet_bootstrap.cc:492] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:31.062333 1267 ts_tablet_manager.cc:1367] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.000s
I20240502 14:00:31.062774 1267 raft_consensus.cc:348] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.062932 1267 raft_consensus.cc:374] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.062979 1267 raft_consensus.cc:729] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.063130 1267 consensus_queue.cc:260] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.063486 1267 ts_tablet_manager.cc:1398] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.093657 29863 consensus_queue.cc:237] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:31.095006 31992 raft_consensus.cc:1261] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.095032 29732 raft_consensus.cc:1261] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.095470 1178 consensus_queue.cc:1035] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.095672 32672 consensus_queue.cc:1035] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.097441 32672 raft_consensus.cc:2932] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.097839 31992 raft_consensus.cc:2932] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.100575 29325 catalog_manager.cc:5484] T f3517a9ff83644ccb5cd4c0ebbd6e929 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.100992 29732 raft_consensus.cc:2932] T f3517a9ff83644ccb5cd4c0ebbd6e929 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.103606 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet f3517a9ff83644ccb5cd4c0ebbd6e929 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:31.104290 1027 tablet_replica.cc:330] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:31.104437 1027 raft_consensus.cc:2229] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.104583 1027 raft_consensus.cc:2258] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.105096 1027 ts_tablet_manager.cc:1875] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:31.106746 1027 ts_tablet_manager.cc:1888] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:31.106810 1027 log.cc:1196] T f3517a9ff83644ccb5cd4c0ebbd6e929 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/f3517a9ff83644ccb5cd4c0ebbd6e929
I20240502 14:00:31.108323 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet f3517a9ff83644ccb5cd4c0ebbd6e929 (table rebalance_test_table_9 [id=69941baeddba4f57a44f698072d21957]) successfully deleted
I20240502 14:00:31.113224 859 consensus_queue.cc:1035] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.116716 32752 consensus_queue.cc:1035] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.135627 644 raft_consensus.cc:482] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.135763 644 raft_consensus.cc:504] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.136276 644 leader_election.cc:288] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:31.136790 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5d56ea730b4f08886cb6298e9786fa" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:31.136942 31861 raft_consensus.cc:2454] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:31.137238 32188 leader_election.cc:302] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:31.137420 644 raft_consensus.cc:2781] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.137480 644 raft_consensus.cc:482] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.137526 644 raft_consensus.cc:3037] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.138417 644 raft_consensus.cc:504] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.138765 644 leader_election.cc:288] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:31.139300 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5d56ea730b4f08886cb6298e9786fa" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:31.139421 31861 raft_consensus.cc:3037] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.140172 31861 raft_consensus.cc:2454] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:31.140448 32188 leader_election.cc:302] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:31.140607 644 raft_consensus.cc:2781] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.140686 644 raft_consensus.cc:686] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:31.140820 644 consensus_queue.cc:237] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.142762 29325 catalog_manager.cc:5484] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:31.143095 31992 raft_consensus.cc:3037] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.145171 1105 consensus_queue.cc:1035] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.149755 1105 consensus_queue.cc:1035] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.151033 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5d56ea730b4f08886cb6298e9786fa" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:31.151062 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5d56ea730b4f08886cb6298e9786fa" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:31.151192 29732 raft_consensus.cc:2454] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:31.199846 32235 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:31.201810 31972 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:31.204169 29580 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:31.211244 31841 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:31.213018 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:31.221194 32101 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:31.225301 29712 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:31.236773 29451 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:31.275436 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:31.279681 31861 raft_consensus.cc:1261] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.280093 1306 consensus_queue.cc:1035] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.280217 29729 raft_consensus.cc:3037] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.281133 29729 raft_consensus.cc:1261] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.281560 1306 consensus_queue.cc:1035] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.283227 29327 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet bd5d56ea730b4f08886cb6298e9786fa (table rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45]). Expected version 1 got 0
I20240502 14:00:31.285143 1116 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:31.285511 1116 tablet.cc:1620] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:31.285771 29327 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet bd5d56ea730b4f08886cb6298e9786fa (table rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45]). Expected version 1 got 0
I20240502 14:00:31.286135 29729 consensus_queue.cc:237] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:31.288309 29862 raft_consensus.cc:1261] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.288635 31861 raft_consensus.cc:1261] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.289075 29322 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet bd5d56ea730b4f08886cb6298e9786fa (table rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45]). Expected version 1 got 0
I20240502 14:00:31.289073 32631 consensus_queue.cc:1035] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.289506 32631 consensus_queue.cc:1035] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.289472 1372 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:31.291375 1371 raft_consensus.cc:2932] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:31.292402 1373 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:31.292450 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45] alter complete (version 1)
I20240502 14:00:31.292775 1373 tablet.cc:1620] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:31.293485 29324 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet bd5d56ea730b4f08886cb6298e9786fa (table rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45]). Expected version 1 got 0
I20240502 14:00:31.293572 31861 raft_consensus.cc:2932] T 209e6922777b49ddb87a3dadf65d651a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:31.294752 29327 catalog_manager.cc:5484] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:31.295202 29862 raft_consensus.cc:2932] T 209e6922777b49ddb87a3dadf65d651a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:31.300220 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 209e6922777b49ddb87a3dadf65d651a with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:31.300449 1027 tablet_replica.cc:330] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:31.300603 1027 raft_consensus.cc:2229] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.300751 1027 raft_consensus.cc:2258] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.301286 1027 ts_tablet_manager.cc:1875] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:31.303395 1027 ts_tablet_manager.cc:1888] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:31.303475 1027 log.cc:1196] T 209e6922777b49ddb87a3dadf65d651a P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/209e6922777b49ddb87a3dadf65d651a
I20240502 14:00:31.303948 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 209e6922777b49ddb87a3dadf65d651a (table rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]) successfully deleted
I20240502 14:00:31.310884 1372 tablet.cc:1620] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:31.314324 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:31.320715 32235 tablet_service.cc:1467] Processing CreateTablet for tablet b0fa578318964569b2e10d7f858acd4d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.321013 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0fa578318964569b2e10d7f858acd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.321177 31841 tablet_service.cc:1467] Processing CreateTablet for tablet b0fa578318964569b2e10d7f858acd4d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.321321 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 9790e7ee6d8945bf85f129ef3f9900ca (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.321440 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0fa578318964569b2e10d7f858acd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.321619 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9790e7ee6d8945bf85f129ef3f9900ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.321799 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 9790e7ee6d8945bf85f129ef3f9900ca (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.322049 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9790e7ee6d8945bf85f129ef3f9900ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.323702 29844 tablet_service.cc:1467] Processing CreateTablet for tablet b0fa578318964569b2e10d7f858acd4d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.323915 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0fa578318964569b2e10d7f858acd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.324074 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 5edd4906c0914f6c83dc2bb1501d15bc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.324290 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edd4906c0914f6c83dc2bb1501d15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.324946 1388 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:31.325711 1267 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:31.326514 1267 tablet_bootstrap.cc:654] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.328725 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 5edd4906c0914f6c83dc2bb1501d15bc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.328955 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edd4906c0914f6c83dc2bb1501d15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.330785 1388 tablet_bootstrap.cc:654] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.330914 1269 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:31.331691 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 9790e7ee6d8945bf85f129ef3f9900ca (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.331970 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9790e7ee6d8945bf85f129ef3f9900ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.332055 1269 tablet_bootstrap.cc:654] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.331691 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 5edd4906c0914f6c83dc2bb1501d15bc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.334681 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edd4906c0914f6c83dc2bb1501d15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.335098 1267 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:31.335183 1267 ts_tablet_manager.cc:1367] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:31.335626 1267 raft_consensus.cc:348] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.335723 1267 raft_consensus.cc:374] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.335748 1267 raft_consensus.cc:729] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.335858 1267 consensus_queue.cc:260] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.336148 1267 ts_tablet_manager.cc:1398] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.336704 1387 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:31.338197 1387 tablet_bootstrap.cc:654] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.338565 1392 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:31.339505 1392 tablet_bootstrap.cc:654] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.340500 1392 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:31.340575 1392 ts_tablet_manager.cc:1367] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:31.340960 1392 raft_consensus.cc:348] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.341053 1392 raft_consensus.cc:374] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.341080 1392 raft_consensus.cc:729] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.341195 1392 consensus_queue.cc:260] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.341442 1392 ts_tablet_manager.cc:1398] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.341960 1392 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:31.342788 1392 tablet_bootstrap.cc:654] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.344421 1392 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:31.344503 1392 ts_tablet_manager.cc:1367] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:31.344942 1392 raft_consensus.cc:348] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.345065 1392 raft_consensus.cc:374] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.345098 1392 raft_consensus.cc:729] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.345229 1392 consensus_queue.cc:260] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.345391 1269 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:31.345479 1269 ts_tablet_manager.cc:1367] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:00:31.345487 1392 ts_tablet_manager.cc:1398] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.345999 1269 raft_consensus.cc:348] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.346112 1269 raft_consensus.cc:374] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.346145 1269 raft_consensus.cc:729] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.346269 1269 consensus_queue.cc:260] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.346552 1269 ts_tablet_manager.cc:1398] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.346710 1269 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:31.347664 1269 tablet_bootstrap.cc:654] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.348798 1269 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:31.348879 1269 ts_tablet_manager.cc:1367] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:31.350016 1269 raft_consensus.cc:348] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.350135 1269 raft_consensus.cc:374] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.350175 1269 raft_consensus.cc:729] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.350297 1269 consensus_queue.cc:260] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.350580 1269 ts_tablet_manager.cc:1398] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:31.354295 1387 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:31.354413 1387 ts_tablet_manager.cc:1367] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 14:00:31.354876 1387 raft_consensus.cc:348] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.355003 1387 raft_consensus.cc:374] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.355031 1387 raft_consensus.cc:729] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.355149 1387 consensus_queue.cc:260] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.355396 1387 ts_tablet_manager.cc:1398] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.355698 1387 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:31.356676 1387 tablet_bootstrap.cc:654] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.360214 1388 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:31.360318 1388 ts_tablet_manager.cc:1367] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240502 14:00:31.360731 1388 raft_consensus.cc:348] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.360826 1388 raft_consensus.cc:374] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.360852 1388 raft_consensus.cc:729] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.360965 1388 consensus_queue.cc:260] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.361387 1388 ts_tablet_manager.cc:1398] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.361667 1388 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:31.361917 1387 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:31.361985 1387 ts_tablet_manager.cc:1367] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:31.362493 1387 raft_consensus.cc:348] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.362589 1387 raft_consensus.cc:374] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.362615 1387 raft_consensus.cc:729] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.362720 1387 consensus_queue.cc:260] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.362967 1387 ts_tablet_manager.cc:1398] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.365823 1388 tablet_bootstrap.cc:654] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.367226 1388 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:31.367324 1388 ts_tablet_manager.cc:1367] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:31.367750 1388 raft_consensus.cc:348] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.367861 1388 raft_consensus.cc:374] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.367893 1388 raft_consensus.cc:729] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.368002 1388 consensus_queue.cc:260] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.368263 1388 ts_tablet_manager.cc:1398] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.399703 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:31.408244 1116 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:31.408617 1116 tablet.cc:1620] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:31.408833 1372 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:31.409174 1372 tablet.cc:1620] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:31.412382 1373 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:31.412714 1373 tablet.cc:1620] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:31.415329 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=f6f5fb6a593a4d9dbac56750fd975a45] alter complete (version 2)
I20240502 14:00:31.455972 858 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.456105 858 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.456570 858 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:31.456957 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fa578318964569b2e10d7f858acd4d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:31.457104 32255 raft_consensus.cc:2454] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:31.457558 31796 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:31.457739 858 raft_consensus.cc:2781] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.457801 858 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.457845 858 raft_consensus.cc:3037] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.458990 858 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.459344 858 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:31.459671 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fa578318964569b2e10d7f858acd4d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:31.459782 32255 raft_consensus.cc:3037] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.460569 32255 raft_consensus.cc:2454] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:31.460856 31796 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:31.461158 858 raft_consensus.cc:2781] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.461246 858 raft_consensus.cc:686] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:31.461390 858 consensus_queue.cc:237] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.461648 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fa578318964569b2e10d7f858acd4d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:31.461807 29862 raft_consensus.cc:2454] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:31.461652 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0fa578318964569b2e10d7f858acd4d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:31.461995 29863 raft_consensus.cc:3037] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.463038 29863 raft_consensus.cc:2454] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:31.463465 29322 catalog_manager.cc:5484] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:31.516604 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:31.522790 1116 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:31.523165 1373 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:31.523516 1372 tablet.cc:1722] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:31.526507 29328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 14:00:31.526635 29328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 14:00:31.529765 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 7bad6684944b481989728378ab51d515
I20240502 14:00:31.530092 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet bda9fdd55bbb40fdb56f5fac6c5a497b
I20240502 14:00:31.530350 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet e227b307a0f44a088bf50899d59f2118
I20240502 14:00:31.530820 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bad6684944b481989728378ab51d515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:31.530910 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet bda9fdd55bbb40fdb56f5fac6c5a497b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:31.531018 1027 tablet_replica.cc:330] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:31.531152 1027 raft_consensus.cc:2229] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.528873 29321 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=f6f5fb6a593a4d9dbac56750fd975a45] alter complete (version 3)
I20240502 14:00:31.531268 1027 raft_consensus.cc:2258] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.531424 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet e227b307a0f44a088bf50899d59f2118 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:31.531756 1027 ts_tablet_manager.cc:1875] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.531917 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet bda9fdd55bbb40fdb56f5fac6c5a497b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:31.532003 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet e227b307a0f44a088bf50899d59f2118 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:31.532155 833 tablet_replica.cc:330] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:31.532284 833 raft_consensus.cc:2229] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.532451 833 raft_consensus.cc:2258] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.532449 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet e227b307a0f44a088bf50899d59f2118 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:31.532616 32460 tablet_replica.cc:330] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:31.532749 32460 raft_consensus.cc:2229] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:31.532927 833 ts_tablet_manager.cc:1875] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.532941 32460 raft_consensus.cc:2258] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.533439 32460 ts_tablet_manager.cc:1875] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.533895 1027 ts_tablet_manager.cc:1888] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.533955 1027 log.cc:1196] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/7bad6684944b481989728378ab51d515
I20240502 14:00:31.534220 1027 ts_tablet_manager.cc:1909] T 7bad6684944b481989728378ab51d515 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:31.534430 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bad6684944b481989728378ab51d515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:31.535312 1430 tablet_replica.cc:330] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:31.535485 1430 raft_consensus.cc:2229] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.535573 32460 ts_tablet_manager.cc:1888] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:00:31.535673 1430 raft_consensus.cc:2258] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.535730 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bad6684944b481989728378ab51d515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:31.535914 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet bda9fdd55bbb40fdb56f5fac6c5a497b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:31.536140 32457 tablet_replica.cc:330] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:31.536154 1430 ts_tablet_manager.cc:1875] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.536293 32457 raft_consensus.cc:2229] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:31.536348 833 ts_tablet_manager.cc:1888] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.536415 833 log.cc:1196] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/bda9fdd55bbb40fdb56f5fac6c5a497b
I20240502 14:00:31.536494 32457 raft_consensus.cc:2258] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.536680 833 ts_tablet_manager.cc:1909] T bda9fdd55bbb40fdb56f5fac6c5a497b P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:31.537042 32457 ts_tablet_manager.cc:1875] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.537438 1027 tablet_replica.cc:330] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:31.537567 1027 raft_consensus.cc:2229] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.537678 1027 raft_consensus.cc:2258] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.538013 1430 ts_tablet_manager.cc:1888] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:00:31.538074 1430 log.cc:1196] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/e227b307a0f44a088bf50899d59f2118
I20240502 14:00:31.538329 1430 ts_tablet_manager.cc:1909] T e227b307a0f44a088bf50899d59f2118 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:31.538411 833 tablet_replica.cc:330] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:31.538522 833 raft_consensus.cc:2229] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.538657 833 raft_consensus.cc:2258] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.539144 1027 ts_tablet_manager.cc:1875] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.539165 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 7bad6684944b481989728378ab51d515 (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.539355 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet e227b307a0f44a088bf50899d59f2118 (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.539965 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet bda9fdd55bbb40fdb56f5fac6c5a497b (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.540637 32457 ts_tablet_manager.cc:1888] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.540807 32460 log.cc:1196] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/e227b307a0f44a088bf50899d59f2118
I20240502 14:00:31.541079 32460 ts_tablet_manager.cc:1909] T e227b307a0f44a088bf50899d59f2118 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:31.541246 1027 ts_tablet_manager.cc:1888] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.541302 1027 log.cc:1196] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/bda9fdd55bbb40fdb56f5fac6c5a497b
I20240502 14:00:31.541560 1027 ts_tablet_manager.cc:1909] T bda9fdd55bbb40fdb56f5fac6c5a497b P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:31.541697 32460 tablet_replica.cc:330] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:31.541797 32460 raft_consensus.cc:2229] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.541918 32460 raft_consensus.cc:2258] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.539167 32672 raft_consensus.cc:482] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.542294 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet bda9fdd55bbb40fdb56f5fac6c5a497b (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.542279 32672 raft_consensus.cc:504] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.542349 833 ts_tablet_manager.cc:1875] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.542739 32672 leader_election.cc:288] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:31.543337 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e7ee6d8945bf85f129ef3f9900ca" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:31.543468 31992 raft_consensus.cc:2454] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:31.543558 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet e227b307a0f44a088bf50899d59f2118 (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.544050 29797 leader_election.cc:302] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:31.544238 32672 raft_consensus.cc:2781] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.544304 32672 raft_consensus.cc:482] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.544348 32672 raft_consensus.cc:3037] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.544369 833 ts_tablet_manager.cc:1888] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:00:31.544430 833 log.cc:1196] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/e227b307a0f44a088bf50899d59f2118
I20240502 14:00:31.544683 833 ts_tablet_manager.cc:1909] T e227b307a0f44a088bf50899d59f2118 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:31.545375 32672 raft_consensus.cc:504] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.545641 32672 leader_election.cc:288] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:31.546012 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e7ee6d8945bf85f129ef3f9900ca" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:31.546072 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet e227b307a0f44a088bf50899d59f2118 (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.546128 31992 raft_consensus.cc:3037] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.546258 32457 log.cc:1196] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/7bad6684944b481989728378ab51d515
I20240502 14:00:31.546517 32457 ts_tablet_manager.cc:1909] T 7bad6684944b481989728378ab51d515 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:31.543277 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e7ee6d8945bf85f129ef3f9900ca" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:31.545917 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e7ee6d8945bf85f129ef3f9900ca" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:31.546897 29470 raft_consensus.cc:2454] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:31.547217 31992 raft_consensus.cc:2454] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:31.547400 32457 tablet_replica.cc:330] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:31.547537 32457 raft_consensus.cc:2229] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:31.547539 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 7bad6684944b481989728378ab51d515 (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.547575 29797 leader_election.cc:302] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:31.547709 32457 raft_consensus.cc:2258] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.547819 32672 raft_consensus.cc:2781] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.547902 32672 raft_consensus.cc:686] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:31.548029 32672 consensus_queue.cc:237] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:31.548450 32460 ts_tablet_manager.cc:1875] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.548940 32457 ts_tablet_manager.cc:1875] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.550820 32457 ts_tablet_manager.cc:1888] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.550936 32457 log.cc:1196] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/bda9fdd55bbb40fdb56f5fac6c5a497b
I20240502 14:00:31.551215 32460 ts_tablet_manager.cc:1888] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:31.551239 32457 ts_tablet_manager.cc:1909] T bda9fdd55bbb40fdb56f5fac6c5a497b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:31.551280 32460 log.cc:1196] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/7bad6684944b481989728378ab51d515
I20240502 14:00:31.551497 32460 ts_tablet_manager.cc:1909] T 7bad6684944b481989728378ab51d515 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:31.551637 29321 catalog_manager.cc:5484] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:31.551921 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet bda9fdd55bbb40fdb56f5fac6c5a497b (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.552274 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 7bad6684944b481989728378ab51d515 (table rebalance_test_table_16 [id=9bc43e10987646f38c64ccd18f0801cf]) successfully deleted
I20240502 14:00:31.564913 32542 raft_consensus.cc:482] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.565053 32542 raft_consensus.cc:504] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.565469 32542 leader_election.cc:288] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:31.565824 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edd4906c0914f6c83dc2bb1501d15bc" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:31.565886 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edd4906c0914f6c83dc2bb1501d15bc" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:31.565948 32255 raft_consensus.cc:2454] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:31.566036 29470 raft_consensus.cc:2454] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:31.566224 31927 leader_election.cc:302] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:31.566375 32542 raft_consensus.cc:2781] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.566429 32542 raft_consensus.cc:482] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.566474 32542 raft_consensus.cc:3037] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.567389 32542 raft_consensus.cc:504] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.567703 32542 leader_election.cc:288] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:31.568009 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edd4906c0914f6c83dc2bb1501d15bc" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:31.568106 32255 raft_consensus.cc:3037] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.568082 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edd4906c0914f6c83dc2bb1501d15bc" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:31.568204 29470 raft_consensus.cc:3037] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.568928 32255 raft_consensus.cc:2454] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:31.569201 29470 raft_consensus.cc:2454] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:31.569232 31927 leader_election.cc:302] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:31.569442 32542 raft_consensus.cc:2781] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.569584 32542 raft_consensus.cc:686] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:31.569726 32542 consensus_queue.cc:237] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:31.571846 29321 catalog_manager.cc:5484] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:31.597589 29451 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:31.597589 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:31.598809 32235 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:31.604878 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:31.606822 32101 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:31.615145 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:31.616351 29582 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:31.630156 29321 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:31.630314 29321 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:31.632668 29321 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet bd5d56ea730b4f08886cb6298e9786fa
I20240502 14:00:31.633298 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5d56ea730b4f08886cb6298e9786fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:31.633411 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5d56ea730b4f08886cb6298e9786fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:31.633481 1430 tablet_replica.cc:330] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:31.633594 1430 raft_consensus.cc:2229] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:31.633715 1445 tablet_replica.cc:330] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:31.633841 1445 raft_consensus.cc:2229] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.633859 1430 raft_consensus.cc:2258] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.633993 1445 raft_consensus.cc:2258] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.634507 1430 ts_tablet_manager.cc:1875] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.634634 1445 ts_tablet_manager.cc:1875] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.634958 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5d56ea730b4f08886cb6298e9786fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:31 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:31.635277 833 tablet_replica.cc:330] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:31.635499 833 raft_consensus.cc:2229] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.635604 833 raft_consensus.cc:2258] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.636190 833 ts_tablet_manager.cc:1875] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:31.636725 1430 ts_tablet_manager.cc:1888] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:31.636788 1430 log.cc:1196] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/bd5d56ea730b4f08886cb6298e9786fa
I20240502 14:00:31.638187 833 ts_tablet_manager.cc:1888] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:31.638264 833 log.cc:1196] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd5d56ea730b4f08886cb6298e9786fa
I20240502 14:00:31.638499 833 ts_tablet_manager.cc:1909] T bd5d56ea730b4f08886cb6298e9786fa P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:31.639322 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet bd5d56ea730b4f08886cb6298e9786fa (table rebalancer_dynamic_table_new_name [id=f6f5fb6a593a4d9dbac56750fd975a45]) successfully deleted
I20240502 14:00:31.643638 1430 ts_tablet_manager.cc:1909] T bd5d56ea730b4f08886cb6298e9786fa P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:31.644241 31841 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:31.644484 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet bd5d56ea730b4f08886cb6298e9786fa (table rebalancer_dynamic_table_new_name [id=f6f5fb6a593a4d9dbac56750fd975a45]) successfully deleted
I20240502 14:00:31.645460 1445 ts_tablet_manager.cc:1888] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:31.645576 1445 log.cc:1196] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/bd5d56ea730b4f08886cb6298e9786fa
I20240502 14:00:31.645902 1445 ts_tablet_manager.cc:1909] T bd5d56ea730b4f08886cb6298e9786fa P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:31.646550 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet bd5d56ea730b4f08886cb6298e9786fa (table rebalancer_dynamic_table_new_name [id=f6f5fb6a593a4d9dbac56750fd975a45]) successfully deleted
I20240502 14:00:31.683081 29732 consensus_queue.cc:237] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:31.684283 29863 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.684319 31992 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.684729 32631 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.684908 1371 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.686549 1080 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.686995 29863 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.689043 31992 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.689388 29321 catalog_manager.cc:5484] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:31.692128 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bdf925a9c4d456fb3b4ef448a2154c6 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:31.692432 1027 tablet_replica.cc:330] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:31.692579 1027 raft_consensus.cc:2229] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.692734 1027 raft_consensus.cc:2258] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.693307 1027 ts_tablet_manager.cc:1875] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:31.695562 1027 ts_tablet_manager.cc:1888] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:31.695633 1027 log.cc:1196] T 9bdf925a9c4d456fb3b4ef448a2154c6 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/9bdf925a9c4d456fb3b4ef448a2154c6
I20240502 14:00:31.696095 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 9bdf925a9c4d456fb3b4ef448a2154c6 (table rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]) successfully deleted
I20240502 14:00:31.735594 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:31.741839 31971 tablet_service.cc:1467] Processing CreateTablet for tablet dd7fc0ae24f94333843b32242850b967 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88]), partition=
I20240502 14:00:31.742009 31841 tablet_service.cc:1467] Processing CreateTablet for tablet dd7fc0ae24f94333843b32242850b967 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88]), partition=
I20240502 14:00:31.742156 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7fc0ae24f94333843b32242850b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.742270 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7fc0ae24f94333843b32242850b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.742237 32235 tablet_service.cc:1467] Processing CreateTablet for tablet dd7fc0ae24f94333843b32242850b967 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88]), partition=
I20240502 14:00:31.742462 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7fc0ae24f94333843b32242850b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.744583 1267 tablet_bootstrap.cc:492] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:31.744886 1387 tablet_bootstrap.cc:492] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:31.745083 1269 tablet_bootstrap.cc:492] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:31.745482 1267 tablet_bootstrap.cc:654] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.745847 1387 tablet_bootstrap.cc:654] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.745937 1269 tablet_bootstrap.cc:654] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.746907 1267 tablet_bootstrap.cc:492] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:31.746989 1267 ts_tablet_manager.cc:1367] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:31.747424 1267 raft_consensus.cc:348] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.747539 1267 raft_consensus.cc:374] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.747576 1267 raft_consensus.cc:729] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.747697 1267 consensus_queue.cc:260] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.747983 1267 ts_tablet_manager.cc:1398] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.750247 1387 tablet_bootstrap.cc:492] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:31.750334 1387 ts_tablet_manager.cc:1367] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:31.750756 1387 raft_consensus.cc:348] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.750887 1387 raft_consensus.cc:374] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.750926 1387 raft_consensus.cc:729] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.751067 1387 consensus_queue.cc:260] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.751339 1387 ts_tablet_manager.cc:1398] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.751412 1269 tablet_bootstrap.cc:492] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:31.751492 1269 ts_tablet_manager.cc:1367] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:31.751876 1269 raft_consensus.cc:348] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.751978 1269 raft_consensus.cc:374] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.752009 1269 raft_consensus.cc:729] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.752146 1269 consensus_queue.cc:260] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.752408 1269 ts_tablet_manager.cc:1398] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.755842 1306 raft_consensus.cc:482] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.755931 1306 raft_consensus.cc:504] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.756274 1306 leader_election.cc:288] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:31.757069 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7fc0ae24f94333843b32242850b967" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:31.757202 31861 raft_consensus.cc:2454] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:31.757782 32188 leader_election.cc:302] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:31.758323 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7fc0ae24f94333843b32242850b967" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:31.758453 31992 raft_consensus.cc:2454] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:31.758486 1306 raft_consensus.cc:2781] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.758531 1306 raft_consensus.cc:482] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.758569 1306 raft_consensus.cc:3037] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.759589 1306 raft_consensus.cc:504] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.759930 1306 leader_election.cc:288] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:31.760329 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7fc0ae24f94333843b32242850b967" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:31.760454 31861 raft_consensus.cc:3037] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.761531 31861 raft_consensus.cc:2454] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:31.762573 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7fc0ae24f94333843b32242850b967" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:31.762627 32188 leader_election.cc:302] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:31.762688 31992 raft_consensus.cc:3037] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.762814 1306 raft_consensus.cc:2781] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.762960 1306 raft_consensus.cc:686] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:31.763090 1306 consensus_queue.cc:237] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.763741 31992 raft_consensus.cc:2454] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:31.765516 29328 catalog_manager.cc:5484] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:31.789096 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:31.791505 32235 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:31.791535 29451 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:31.794090 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:31.801112 29582 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:31.808306 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:31.820919 31841 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:31.834578 32101 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:31.868995 29324 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:31.873337 31992 raft_consensus.cc:1261] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.873414 31861 raft_consensus.cc:1261] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.873894 1306 consensus_queue.cc:1035] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.874449 29601 consensus_queue.cc:237] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:31.875288 1369 consensus_queue.cc:1035] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.876164 32255 raft_consensus.cc:1261] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.876216 29470 raft_consensus.cc:1261] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:31.876572 1537 consensus_queue.cc:1035] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.876821 1132 consensus_queue.cc:1035] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:31.877146 29324 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet dd7fc0ae24f94333843b32242850b967 (table rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88]). Expected version 1 got 0
I20240502 14:00:31.879161 1116 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:31.879488 1116 tablet.cc:1620] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:31.880425 1373 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:31.880718 1373 tablet.cc:1620] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:31.880751 1132 raft_consensus.cc:2932] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:31.882802 29470 raft_consensus.cc:2932] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:31.884764 29324 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet dd7fc0ae24f94333843b32242850b967 (table rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88]). Expected version 1 got 0
I20240502 14:00:31.885644 29325 catalog_manager.cc:5484] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:31.891606 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet ba65f51fa3f64a479e1e62042c1ca64e with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:31.891618 29321 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88] alter complete (version 1)
I20240502 14:00:31.891861 833 tablet_replica.cc:330] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:31.892009 833 raft_consensus.cc:2229] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:31.892143 833 raft_consensus.cc:2258] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:31.892738 833 ts_tablet_manager.cc:1875] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:31.893385 29328 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet dd7fc0ae24f94333843b32242850b967 (table rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88]). Expected version 1 got 0
I20240502 14:00:31.895177 833 ts_tablet_manager.cc:1888] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:31.895241 833 log.cc:1196] T ba65f51fa3f64a479e1e62042c1ca64e P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba65f51fa3f64a479e1e62042c1ca64e
I20240502 14:00:31.895766 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet ba65f51fa3f64a479e1e62042c1ca64e (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:31.895898 1546 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:31.896298 1546 tablet.cc:1620] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:31.900257 32255 raft_consensus.cc:2932] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:31.946787 858 consensus_queue.cc:1035] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.952899 1075 consensus_queue.cc:1035] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.985777 29451 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:31.994330 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:31.997030 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:31.997692 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:32.005002 1116 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:32.005422 1116 tablet.cc:1620] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:32.006203 1546 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:32.006490 1546 tablet.cc:1620] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:32.009892 32101 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:32.012995 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=66b98152c711486aac60d952e6d29b88] alter complete (version 2)
I20240502 14:00:32.014683 1373 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:32.014990 1373 tablet.cc:1620] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:32.017210 32235 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:32.021215 31841 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:32.024466 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:32.031881 29582 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:32.032483 29320 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 14:00:32.032627 29320 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 14:00:32.035373 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet e6f7a8f5b0b047208246d639ccf24447
I20240502 14:00:32.035720 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 8856fc8643184e73bf7156c1186b4b80
I20240502 14:00:32.035890 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet ba65f51fa3f64a479e1e62042c1ca64e
I20240502 14:00:32.035910 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f7a8f5b0b047208246d639ccf24447 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:32.036082 1027 tablet_replica.cc:330] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:32.036242 1027 raft_consensus.cc:2229] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.036361 1027 raft_consensus.cc:2258] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.036880 1027 ts_tablet_manager.cc:1875] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.037663 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet ba65f51fa3f64a479e1e62042c1ca64e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:32.037846 32457 tablet_replica.cc:330] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:32.037950 32457 raft_consensus.cc:2229] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.038098 32457 raft_consensus.cc:2258] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.038411 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet ba65f51fa3f64a479e1e62042c1ca64e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:32.038569 1430 tablet_replica.cc:330] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:32.038671 1430 raft_consensus.cc:2229] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.038774 1430 raft_consensus.cc:2258] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.038987 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f7a8f5b0b047208246d639ccf24447 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:32.039103 1027 ts_tablet_manager.cc:1888] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:32.039162 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 8856fc8643184e73bf7156c1186b4b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:32.039187 1027 log.cc:1196] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/e6f7a8f5b0b047208246d639ccf24447
I20240502 14:00:32.039439 1027 ts_tablet_manager.cc:1909] T e6f7a8f5b0b047208246d639ccf24447 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:32.040244 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 8856fc8643184e73bf7156c1186b4b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:32.040400 32460 tablet_replica.cc:330] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:32.040508 32460 raft_consensus.cc:2229] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:32.040674 32460 raft_consensus.cc:2258] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.041141 32460 ts_tablet_manager.cc:1875] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.043537 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet e6f7a8f5b0b047208246d639ccf24447 (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.043609 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet ba65f51fa3f64a479e1e62042c1ca64e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:32.043898 1027 tablet_replica.cc:330] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:32.044128 1027 raft_consensus.cc:2229] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:32.044410 1027 raft_consensus.cc:2258] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.045531 1027 ts_tablet_manager.cc:1875] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.045555 1430 ts_tablet_manager.cc:1875] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.047757 1430 ts_tablet_manager.cc:1888] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:00:32.047839 1430 log.cc:1196] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/ba65f51fa3f64a479e1e62042c1ca64e
I20240502 14:00:32.047946 1027 ts_tablet_manager.cc:1888] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:00:32.048012 1027 log.cc:1196] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba65f51fa3f64a479e1e62042c1ca64e
I20240502 14:00:32.048132 1430 ts_tablet_manager.cc:1909] T ba65f51fa3f64a479e1e62042c1ca64e P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:32.048301 1027 ts_tablet_manager.cc:1909] T ba65f51fa3f64a479e1e62042c1ca64e P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:32.049782 32457 ts_tablet_manager.cc:1875] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.051939 32457 ts_tablet_manager.cc:1888] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 14:00:32.052002 32457 log.cc:1196] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba65f51fa3f64a479e1e62042c1ca64e
I20240502 14:00:32.052253 32457 ts_tablet_manager.cc:1909] T ba65f51fa3f64a479e1e62042c1ca64e P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:32.052908 32457 tablet_replica.cc:330] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:32.053040 32457 raft_consensus.cc:2229] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.053160 32457 raft_consensus.cc:2258] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.053680 32457 ts_tablet_manager.cc:1875] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.053663 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet ba65f51fa3f64a479e1e62042c1ca64e (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.055441 32457 ts_tablet_manager.cc:1888] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:32.055507 32457 log.cc:1196] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/e6f7a8f5b0b047208246d639ccf24447
I20240502 14:00:32.055760 32457 ts_tablet_manager.cc:1909] T e6f7a8f5b0b047208246d639ccf24447 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:32.055770 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet ba65f51fa3f64a479e1e62042c1ca64e (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.056386 32457 tablet_replica.cc:330] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:32.056499 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f7a8f5b0b047208246d639ccf24447 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:32.056551 32457 raft_consensus.cc:2229] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.056614 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet 8856fc8643184e73bf7156c1186b4b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:32.056670 833 tablet_replica.cc:330] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:32.056680 32457 raft_consensus.cc:2258] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.056802 833 raft_consensus.cc:2229] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:32.056968 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet e6f7a8f5b0b047208246d639ccf24447 (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.057003 833 raft_consensus.cc:2258] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.057163 32457 ts_tablet_manager.cc:1875] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.057497 833 ts_tablet_manager.cc:1875] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.058336 32460 ts_tablet_manager.cc:1888] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:32.058414 32460 log.cc:1196] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/8856fc8643184e73bf7156c1186b4b80
I20240502 14:00:32.058694 32460 ts_tablet_manager.cc:1909] T 8856fc8643184e73bf7156c1186b4b80 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:32.058969 32457 ts_tablet_manager.cc:1888] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:32.059026 32457 log.cc:1196] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/8856fc8643184e73bf7156c1186b4b80
I20240502 14:00:32.059269 32457 ts_tablet_manager.cc:1909] T 8856fc8643184e73bf7156c1186b4b80 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:32.059707 833 ts_tablet_manager.cc:1888] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:32.059775 833 log.cc:1196] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6f7a8f5b0b047208246d639ccf24447
I20240502 14:00:32.059789 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 8856fc8643184e73bf7156c1186b4b80 (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.059988 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 8856fc8643184e73bf7156c1186b4b80 (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.060034 833 ts_tablet_manager.cc:1909] T e6f7a8f5b0b047208246d639ccf24447 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:32.060560 833 tablet_replica.cc:330] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:32.060675 833 raft_consensus.cc:2229] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.060710 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet e6f7a8f5b0b047208246d639ccf24447 (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.060745 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet ba65f51fa3f64a479e1e62042c1ca64e (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.060788 833 raft_consensus.cc:2258] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.061218 833 ts_tablet_manager.cc:1875] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.064097 833 ts_tablet_manager.cc:1888] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:00:32.064168 833 log.cc:1196] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/8856fc8643184e73bf7156c1186b4b80
I20240502 14:00:32.064414 833 ts_tablet_manager.cc:1909] T 8856fc8643184e73bf7156c1186b4b80 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:32.065043 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 8856fc8643184e73bf7156c1186b4b80 (table rebalance_test_table_18 [id=d2e9f2fea88446a9ba35aea5ecbf43ba]) successfully deleted
I20240502 14:00:32.084645 32672 consensus_queue.cc:1035] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.087370 29324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:32.088655 29470 raft_consensus.cc:3037] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.090286 1178 consensus_queue.cc:1035] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.094523 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 04f0f9efcbb544d7b8292158bf769426 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.094885 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f0f9efcbb544d7b8292158bf769426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.094986 32233 tablet_service.cc:1467] Processing CreateTablet for tablet a8e5364d50da4cfb8fbbf21913324bc6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.095255 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e5364d50da4cfb8fbbf21913324bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.095312 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 04f0f9efcbb544d7b8292158bf769426 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.095561 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f0f9efcbb544d7b8292158bf769426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.097157 1269 tablet_bootstrap.cc:492] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:32.097184 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 04f0f9efcbb544d7b8292158bf769426 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.097442 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f0f9efcbb544d7b8292158bf769426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.097712 1593 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:32.098024 1269 tablet_bootstrap.cc:654] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.098484 29449 tablet_service.cc:1467] Processing CreateTablet for tablet f8b939db0e5b464dbcecbf8e18cccadf (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.098702 1593 tablet_bootstrap.cc:654] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.098731 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b939db0e5b464dbcecbf8e18cccadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.099485 1387 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:32.099599 31841 tablet_service.cc:1467] Processing CreateTablet for tablet a8e5364d50da4cfb8fbbf21913324bc6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.099864 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e5364d50da4cfb8fbbf21913324bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.100067 32103 tablet_service.cc:1467] Processing CreateTablet for tablet f8b939db0e5b464dbcecbf8e18cccadf (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.100306 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b939db0e5b464dbcecbf8e18cccadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.100356 1387 tablet_bootstrap.cc:654] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.101485 1387 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:32.101459 32101 tablet_service.cc:1467] Processing CreateTablet for tablet a8e5364d50da4cfb8fbbf21913324bc6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.101562 1387 ts_tablet_manager.cc:1367] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:32.101651 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e5364d50da4cfb8fbbf21913324bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.101997 1387 raft_consensus.cc:348] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.102095 1387 raft_consensus.cc:374] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.102141 1387 raft_consensus.cc:729] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.102110 31840 tablet_service.cc:1467] Processing CreateTablet for tablet f8b939db0e5b464dbcecbf8e18cccadf (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.102257 1387 consensus_queue.cc:260] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.102326 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b939db0e5b464dbcecbf8e18cccadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.102528 1387 ts_tablet_manager.cc:1398] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.104167 1267 tablet_bootstrap.cc:492] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:32.104897 1267 tablet_bootstrap.cc:654] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.106146 1267 tablet_bootstrap.cc:492] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:32.106253 1267 ts_tablet_manager.cc:1367] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:32.106426 1593 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:32.106484 1593 ts_tablet_manager.cc:1367] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:32.106666 1267 raft_consensus.cc:348] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.106760 1267 raft_consensus.cc:374] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.106787 1267 raft_consensus.cc:729] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.106809 1593 raft_consensus.cc:348] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.106930 1593 raft_consensus.cc:374] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.106966 1593 raft_consensus.cc:729] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.106925 1267 consensus_queue.cc:260] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.107070 1593 consensus_queue.cc:260] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.107187 1267 ts_tablet_manager.cc:1398] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.107332 1593 ts_tablet_manager.cc:1398] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.107338 1267 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:32.107473 1593 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:32.108386 1593 tablet_bootstrap.cc:654] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.110116 1267 tablet_bootstrap.cc:654] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.111562 1269 tablet_bootstrap.cc:492] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:32.111660 1269 ts_tablet_manager.cc:1367] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:00:32.111675 1593 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:32.111743 1593 ts_tablet_manager.cc:1367] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:32.112099 1269 raft_consensus.cc:348] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.112161 1267 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:32.112190 1269 raft_consensus.cc:374] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.112210 1269 raft_consensus.cc:729] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.112179 1593 raft_consensus.cc:348] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.112241 1267 ts_tablet_manager.cc:1367] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:32.112270 1593 raft_consensus.cc:374] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.112300 1593 raft_consensus.cc:729] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.112330 1269 consensus_queue.cc:260] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.112437 1593 consensus_queue.cc:260] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.112609 1269 ts_tablet_manager.cc:1398] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.112767 1269 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:32.112742 1267 raft_consensus.cc:348] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.112833 1267 raft_consensus.cc:374] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.112867 1267 raft_consensus.cc:729] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.112988 1267 consensus_queue.cc:260] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.113248 1267 ts_tablet_manager.cc:1398] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:32.113647 1269 tablet_bootstrap.cc:654] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.112699 1593 ts_tablet_manager.cc:1398] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:32.115468 1269 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:32.115557 1269 ts_tablet_manager.cc:1367] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:32.116133 1269 raft_consensus.cc:348] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.116331 1269 raft_consensus.cc:374] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.116410 1269 raft_consensus.cc:729] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.116461 1593 tablet_bootstrap.cc:492] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:32.116621 1269 consensus_queue.cc:260] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.117149 1269 ts_tablet_manager.cc:1398] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.117446 1596 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:32.117391 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:32.118428 1596 tablet_bootstrap.cc:654] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.118970 1593 tablet_bootstrap.cc:654] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.121583 1593 tablet_bootstrap.cc:492] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:32.121659 1593 ts_tablet_manager.cc:1367] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 14:00:32.121868 1596 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:32.121949 1596 ts_tablet_manager.cc:1367] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:32.122463 1596 raft_consensus.cc:348] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.122570 1596 raft_consensus.cc:374] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.122529 1593 raft_consensus.cc:348] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.122599 1596 raft_consensus.cc:729] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.122612 1593 raft_consensus.cc:374] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.122645 1593 raft_consensus.cc:729] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.122735 1596 consensus_queue.cc:260] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.122745 1593 consensus_queue.cc:260] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.123124 1596 ts_tablet_manager.cc:1398] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:32.123725 1593 ts_tablet_manager.cc:1398] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:32.126243 1116 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:32.127087 1546 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:32.127171 1373 tablet.cc:1722] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:32.131788 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=66b98152c711486aac60d952e6d29b88] alter complete (version 3)
I20240502 14:00:32.135392 32542 consensus_queue.cc:1035] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.141398 32542 consensus_queue.cc:1035] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.146425 1369 raft_consensus.cc:482] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.146513 1369 raft_consensus.cc:504] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.146929 1369 leader_election.cc:288] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:32.147500 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f0f9efcbb544d7b8292158bf769426" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:32.147621 31990 raft_consensus.cc:2454] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:32.147997 32188 leader_election.cc:302] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:32.148430 1369 raft_consensus.cc:2781] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.148480 1369 raft_consensus.cc:482] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.148514 1369 raft_consensus.cc:3037] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.149480 1369 raft_consensus.cc:504] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.149757 1369 leader_election.cc:288] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:32.150287 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f0f9efcbb544d7b8292158bf769426" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:32.150792 31990 raft_consensus.cc:3037] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.151738 31990 raft_consensus.cc:2454] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:32.152009 32188 leader_election.cc:302] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:32.152192 1369 raft_consensus.cc:2781] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.152288 1369 raft_consensus.cc:686] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:32.152451 1369 consensus_queue.cc:237] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.154464 29323 catalog_manager.cc:5484] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:32.157343 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f0f9efcbb544d7b8292158bf769426" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:32.157390 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f0f9efcbb544d7b8292158bf769426" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:32.157469 32119 raft_consensus.cc:2454] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:32.160655 32122 consensus_queue.cc:237] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:32.162042 32254 raft_consensus.cc:1261] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.162303 29470 raft_consensus.cc:1261] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:32.162312 32057 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82cda992fe784d9f9015e6a3d431f1cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.162714 1626 consensus_queue.cc:1035] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.163024 32752 consensus_queue.cc:1035] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.164808 32752 raft_consensus.cc:2932] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.165243 29470 raft_consensus.cc:2932] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.167055 32254 raft_consensus.cc:2932] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.167732 29323 catalog_manager.cc:5484] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.186673 29732 consensus_queue.cc:237] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.188551 29863 raft_consensus.cc:1261] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:32.188606 29601 raft_consensus.cc:1261] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:32.188925 1080 consensus_queue.cc:1035] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:32.189110 1646 consensus_queue.cc:1035] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:32.191002 1646 raft_consensus.cc:2932] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.191350 29863 raft_consensus.cc:2932] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.193969 29329 catalog_manager.cc:5484] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
W20240502 14:00:32.194944 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93e77c40b5674865ba6ccce626545ec7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.195312 29601 raft_consensus.cc:2932] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.235185 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:32.235349 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:32.237505 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet dd7fc0ae24f94333843b32242850b967
I20240502 14:00:32.238070 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7fc0ae24f94333843b32242850b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:32.238190 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7fc0ae24f94333843b32242850b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:32.238384 1430 tablet_replica.cc:330] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:32.238453 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7fc0ae24f94333843b32242850b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:32.238546 1430 raft_consensus.cc:2229] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:32.238754 1430 raft_consensus.cc:2258] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.240063 1430 ts_tablet_manager.cc:1875] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.242192 1430 ts_tablet_manager.cc:1888] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:32.242260 1430 log.cc:1196] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/dd7fc0ae24f94333843b32242850b967
I20240502 14:00:32.242532 1430 ts_tablet_manager.cc:1909] T dd7fc0ae24f94333843b32242850b967 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:32.245599 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet dd7fc0ae24f94333843b32242850b967 (table rebalancer_dynamic_table_new_name [id=66b98152c711486aac60d952e6d29b88]) successfully deleted
I20240502 14:00:32.246537 1685 tablet_replica.cc:330] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:32.246703 1685 raft_consensus.cc:2229] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.247047 1684 tablet_replica.cc:330] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:32.247251 1684 raft_consensus.cc:2229] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.247457 1685 raft_consensus.cc:2258] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.248667 1625 raft_consensus.cc:482] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.248771 1625 raft_consensus.cc:504] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.249228 1625 leader_election.cc:288] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:32.249580 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5364d50da4cfb8fbbf21913324bc6" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:32.249625 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5364d50da4cfb8fbbf21913324bc6" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:32.249702 32254 raft_consensus.cc:2454] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:32.249728 31861 raft_consensus.cc:2454] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:32.250021 32058 leader_election.cc:302] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:32.250202 1625 raft_consensus.cc:2781] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.250258 1625 raft_consensus.cc:482] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.250296 1625 raft_consensus.cc:3037] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.251329 1625 raft_consensus.cc:504] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.247438 1684 raft_consensus.cc:2258] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.251691 1625 leader_election.cc:288] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:32.252242 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5364d50da4cfb8fbbf21913324bc6" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:32.252236 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5364d50da4cfb8fbbf21913324bc6" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:32.252337 31861 raft_consensus.cc:3037] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.252351 32254 raft_consensus.cc:3037] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.252344 1684 ts_tablet_manager.cc:1875] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.253212 31861 raft_consensus.cc:2454] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:32.253212 32254 raft_consensus.cc:2454] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:32.253619 32056 leader_election.cc:302] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:32.255148 1625 raft_consensus.cc:2781] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.255239 1625 raft_consensus.cc:686] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:32.255370 1625 consensus_queue.cc:237] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.255636 1684 ts_tablet_manager.cc:1888] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:32.255703 1684 log.cc:1196] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/dd7fc0ae24f94333843b32242850b967
I20240502 14:00:32.255983 1684 ts_tablet_manager.cc:1909] T dd7fc0ae24f94333843b32242850b967 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:32.256718 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet dd7fc0ae24f94333843b32242850b967 (table rebalancer_dynamic_table_new_name [id=66b98152c711486aac60d952e6d29b88]) successfully deleted
I20240502 14:00:32.257265 1685 ts_tablet_manager.cc:1875] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.257805 29325 catalog_manager.cc:5484] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.259605 1685 ts_tablet_manager.cc:1888] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:32.259722 1685 log.cc:1196] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/dd7fc0ae24f94333843b32242850b967
I20240502 14:00:32.259975 1685 ts_tablet_manager.cc:1909] T dd7fc0ae24f94333843b32242850b967 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:32.261390 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet dd7fc0ae24f94333843b32242850b967 (table rebalancer_dynamic_table_new_name [id=66b98152c711486aac60d952e6d29b88]) successfully deleted
I20240502 14:00:32.295548 1626 raft_consensus.cc:482] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.295691 1626 raft_consensus.cc:504] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.296156 1626 leader_election.cc:288] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:32.296643 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b939db0e5b464dbcecbf8e18cccadf" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:32.296785 29470 raft_consensus.cc:2454] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:32.297159 32058 leader_election.cc:302] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:32.297369 1626 raft_consensus.cc:2781] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.297371 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b939db0e5b464dbcecbf8e18cccadf" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:32.297425 1626 raft_consensus.cc:482] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.297461 1626 raft_consensus.cc:3037] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.297498 31861 raft_consensus.cc:2454] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:32.298589 1626 raft_consensus.cc:504] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.298975 1626 leader_election.cc:288] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:32.299608 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b939db0e5b464dbcecbf8e18cccadf" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:32.299719 29470 raft_consensus.cc:3037] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.300112 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b939db0e5b464dbcecbf8e18cccadf" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:32.300211 31861 raft_consensus.cc:3037] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.300709 29470 raft_consensus.cc:2454] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:32.301152 31861 raft_consensus.cc:2454] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:32.301424 32056 leader_election.cc:302] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:32.301604 1626 raft_consensus.cc:2781] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.301698 1626 raft_consensus.cc:686] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:32.301837 1626 consensus_queue.cc:237] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.303517 29329 catalog_manager.cc:5484] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.333220 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:32.334374 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:32.341012 29329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:32.345718 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:32.346338 29582 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:32.355926 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:32.357587 32101 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:32.357789 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffa686c8a8e47a5bf770e236b915b24 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=937cd3d02e71476c8c5c7081919f1fd8]), partition=
I20240502 14:00:32.358094 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffa686c8a8e47a5bf770e236b915b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.358186 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffa686c8a8e47a5bf770e236b915b24 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=937cd3d02e71476c8c5c7081919f1fd8]), partition=
I20240502 14:00:32.358409 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffa686c8a8e47a5bf770e236b915b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.360495 1593 tablet_bootstrap.cc:492] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:32.361441 1593 tablet_bootstrap.cc:654] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.362140 1700 tablet_bootstrap.cc:492] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:32.362166 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffa686c8a8e47a5bf770e236b915b24 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=937cd3d02e71476c8c5c7081919f1fd8]), partition=
I20240502 14:00:32.362370 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffa686c8a8e47a5bf770e236b915b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.363078 1700 tablet_bootstrap.cc:654] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.364509 1267 tablet_bootstrap.cc:492] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:32.365305 1267 tablet_bootstrap.cc:654] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.365783 1700 tablet_bootstrap.cc:492] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:32.365856 1700 ts_tablet_manager.cc:1367] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:32.366223 1700 raft_consensus.cc:348] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.366339 1700 raft_consensus.cc:374] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.366369 1700 raft_consensus.cc:729] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.366472 1700 consensus_queue.cc:260] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.366726 1700 ts_tablet_manager.cc:1398] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:32.367669 1267 tablet_bootstrap.cc:492] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:32.367765 1267 ts_tablet_manager.cc:1367] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:32.368253 1267 raft_consensus.cc:348] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.368402 1267 raft_consensus.cc:374] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.368434 1267 raft_consensus.cc:729] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.368546 1267 consensus_queue.cc:260] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.368790 1267 ts_tablet_manager.cc:1398] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.369196 32235 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:32.377703 31840 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:32.382550 1593 tablet_bootstrap.cc:492] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:32.382658 1593 ts_tablet_manager.cc:1367] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240502 14:00:32.383198 1593 raft_consensus.cc:348] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.383324 1593 raft_consensus.cc:374] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.383354 1593 raft_consensus.cc:729] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.383468 1593 consensus_queue.cc:260] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.384111 1593 ts_tablet_manager.cc:1398] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.385437 1075 raft_consensus.cc:482] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.385542 1075 raft_consensus.cc:504] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.385967 1075 leader_election.cc:288] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:32.386474 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa686c8a8e47a5bf770e236b915b24" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:32.386519 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa686c8a8e47a5bf770e236b915b24" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:32.386601 29601 raft_consensus.cc:2454] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:32.386631 32122 raft_consensus.cc:2454] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:32.387065 31794 leader_election.cc:302] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:32.387223 1075 raft_consensus.cc:2781] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.387264 1075 raft_consensus.cc:482] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.387297 1075 raft_consensus.cc:3037] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.388319 1075 raft_consensus.cc:504] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.388625 1075 leader_election.cc:288] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:32.390050 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa686c8a8e47a5bf770e236b915b24" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:32.390151 32122 raft_consensus.cc:3037] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.391098 32122 raft_consensus.cc:2454] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:32.391355 31794 leader_election.cc:302] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:32.391865 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa686c8a8e47a5bf770e236b915b24" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:32.391996 29601 raft_consensus.cc:3037] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.392938 29601 raft_consensus.cc:2454] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:32.393978 1075 raft_consensus.cc:2781] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.394061 1075 raft_consensus.cc:686] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:32.394181 1075 consensus_queue.cc:237] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.397913 29327 catalog_manager.cc:5484] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.450516 31990 consensus_queue.cc:237] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:32.452067 31861 raft_consensus.cc:1261] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.452123 32122 raft_consensus.cc:1261] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.452526 1721 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.452877 1610 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:00:32.453052 31926 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe523f0c364f447cb2a62b599d030cca. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.455215 1610 raft_consensus.cc:2932] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.455646 32122 raft_consensus.cc:2932] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.457006 31861 raft_consensus.cc:2932] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.458674 29327 catalog_manager.cc:5484] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.476778 29863 consensus_queue.cc:237] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:32.478289 29470 raft_consensus.cc:1261] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:32.478330 29601 raft_consensus.cc:1261] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:32.478451 29799 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e41bff5f1484f008851e76cc095eb91. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.478675 1740 consensus_queue.cc:1035] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:32.478914 32672 consensus_queue.cc:1035] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:32.480607 1740 raft_consensus.cc:2932] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:32.480978 29471 raft_consensus.cc:2932] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:32.483441 29601 raft_consensus.cc:2932] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:32.484262 29324 catalog_manager.cc:5484] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:32.506951 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:32.508713 29471 consensus_queue.cc:237] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:32.510730 29732 raft_consensus.cc:1261] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:32.511296 29863 raft_consensus.cc:1261] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:32.511698 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeaf1db0298741fa813660a8e38d01f9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.512140 1105 consensus_queue.cc:1035] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:32.512190 32122 raft_consensus.cc:1261] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.512296 29601 raft_consensus.cc:1261] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.512495 1105 consensus_queue.cc:1035] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:32.512734 858 consensus_queue.cc:1035] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.513027 858 consensus_queue.cc:1035] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.516291 29329 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 1ffa686c8a8e47a5bf770e236b915b24 (table rebalancer_dynamic_table [id=937cd3d02e71476c8c5c7081919f1fd8]). Expected version 1 got 0
I20240502 14:00:32.516497 1699 raft_consensus.cc:2932] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:32.516903 29863 raft_consensus.cc:2932] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:32.517518 29327 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 1ffa686c8a8e47a5bf770e236b915b24 (table rebalancer_dynamic_table [id=937cd3d02e71476c8c5c7081919f1fd8]). Expected version 1 got 0
I20240502 14:00:32.517896 1373 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:32.518247 1373 tablet.cc:1620] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:32.519364 29732 raft_consensus.cc:2932] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:32.522328 29328 catalog_manager.cc:5484] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:32.522814 1768 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:32.523139 1768 tablet.cc:1620] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:32.525610 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=937cd3d02e71476c8c5c7081919f1fd8] alter complete (version 1)
I20240502 14:00:32.529675 1760 mvcc.cc:205] Tried to move back new op lower bound from 7023240939565518848 to 7023240939087847424. Current Snapshot: MvccSnapshot[applied={T|T < 7023240939565518848}]
I20240502 14:00:32.530036 1770 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:32.530380 1770 tablet.cc:1620] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:32.549283 29863 consensus_queue.cc:237] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.550935 31990 raft_consensus.cc:1261] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.551254 29471 raft_consensus.cc:1261] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:32.551303 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9790e7ee6d8945bf85f129ef3f9900ca. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.551590 1740 consensus_queue.cc:1035] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.551733 1178 consensus_queue.cc:1035] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.553252 1178 raft_consensus.cc:2932] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.553740 31990 raft_consensus.cc:2932] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.556470 29324 catalog_manager.cc:5484] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.558502 29470 raft_consensus.cc:2932] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:32.576239 31861 consensus_queue.cc:237] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.577975 31990 raft_consensus.cc:1261] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.578063 32254 raft_consensus.cc:1261] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:32.578325 31795 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27e9d8ef4f754df3817cc4a981549e02. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.578474 1075 consensus_queue.cc:1035] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.578634 858 consensus_queue.cc:1035] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.580513 1075 raft_consensus.cc:2932] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:32.582186 31990 raft_consensus.cc:2932] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:32.582334 32254 raft_consensus.cc:2932] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:32.584746 29324 catalog_manager.cc:5484] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:32.604110 29601 consensus_queue.cc:237] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:32.605461 31990 raft_consensus.cc:1261] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:32.605741 29536 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9f48a525d8c4a0783f0450aa6ec7cb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.605779 32254 raft_consensus.cc:1261] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.606065 1132 consensus_queue.cc:1035] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.606297 1822 consensus_queue.cc:1035] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.608608 1132 raft_consensus.cc:2932] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.608903 31990 raft_consensus.cc:2932] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.611267 32254 raft_consensus.cc:2932] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.611716 29323 catalog_manager.cc:5484] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:32.629741 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:32.633795 32122 consensus_queue.cc:237] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:32.635418 31861 raft_consensus.cc:1261] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.635449 32254 raft_consensus.cc:1261] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.635844 1625 consensus_queue.cc:1035] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.636052 1626 consensus_queue.cc:1035] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:32.636447 1373 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:32.636790 1373 tablet.cc:1620] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:32.636852 1768 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:32.637115 1768 tablet.cc:1620] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:32.637282 1770 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
W20240502 14:00:32.637403 32058 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70367c7c718d4d9195f0af7e69bdf2ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:32.637566 1770 tablet.cc:1620] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:32.639611 1626 raft_consensus.cc:2932] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:32.642709 31861 raft_consensus.cc:2932] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:32.642972 29323 catalog_manager.cc:5484] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:32.645047 32254 raft_consensus.cc:2932] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:32.646582 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=937cd3d02e71476c8c5c7081919f1fd8] alter complete (version 2)
I20240502 14:00:32.739998 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:32.742695 1897 ts_tablet_manager.cc:902] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:32.743392 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:32.743697 1897 tablet_copy_client.cc:308] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:32.744606 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 82cda992fe784d9f9015e6a3d431f1cd from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:50893)
I20240502 14:00:32.744745 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 82cda992fe784d9f9015e6a3d431f1cd from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:50893: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-82cda992fe784d9f9015e6a3d431f1cd
I20240502 14:00:32.746131 32143 tablet_copy_source_session.cc:205] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:32.747491 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:32.747821 32101 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:32.747885 32235 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:32.751295 1897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cda992fe784d9f9015e6a3d431f1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.753763 1897 tablet_copy_client.cc:783] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:32.755802 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:32.759462 1897 tablet_copy_client.cc:647] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:32.760383 29582 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:32.763800 1373 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:32.775997 1306 consensus_queue.cc:1035] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.781163 1625 consensus_queue.cc:1035] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.782970 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=937cd3d02e71476c8c5c7081919f1fd8] alter complete (version 3)
I20240502 14:00:32.783353 1768 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:32.784626 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:32.784798 32752 consensus_queue.cc:1035] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.785936 1770 tablet.cc:1722] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:32.787743 32119 raft_consensus.cc:3037] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.797219 1904 consensus_queue.cc:1035] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.797775 1902 ts_tablet_manager.cc:902] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:32.799620 1897 tablet_copy_client.cc:523] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:32.801530 1897 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:32.804323 32752 consensus_queue.cc:1035] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.805058 31840 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:32.808641 1897 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:32.808949 1897 tablet_bootstrap.cc:492] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:00:32.809139 1897 ts_tablet_manager.cc:1367] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 14:00:32.809661 1897 raft_consensus.cc:348] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:32.809785 1897 raft_consensus.cc:729] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.809957 1897 consensus_queue.cc:260] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:32.810729 1897 ts_tablet_manager.cc:1398] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:00:32.811291 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-82cda992fe784d9f9015e6a3d431f1cd received from {username='slave'} at 127.22.55.132:50893
I20240502 14:00:32.811390 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-82cda992fe784d9f9015e6a3d431f1cd on tablet 82cda992fe784d9f9015e6a3d431f1cd with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:32.814790 1902 tablet_copy_client.cc:308] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:32.815430 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 93e77c40b5674865ba6ccce626545ec7 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:55687)
I20240502 14:00:32.815552 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 93e77c40b5674865ba6ccce626545ec7 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:55687: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-93e77c40b5674865ba6ccce626545ec7
I20240502 14:00:32.816437 32752 consensus_queue.cc:1035] T a8e5364d50da4cfb8fbbf21913324bc6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.816533 29753 tablet_copy_source_session.cc:205] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:32.817651 1902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e77c40b5674865ba6ccce626545ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.819133 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:32.821172 1902 tablet_copy_client.cc:783] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:32.821341 1902 tablet_copy_client.cc:647] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:32.822825 1902 tablet_copy_client.cc:523] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:32.824496 1902 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:32.827209 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce0cb93decb4313a1677fd5613d1749 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.827229 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce0cb93decb4313a1677fd5613d1749 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.827502 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce0cb93decb4313a1677fd5613d1749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.827567 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce0cb93decb4313a1677fd5613d1749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.828904 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2ab852e31d4181a8597b9047a08554 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.829185 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2ab852e31d4181a8597b9047a08554. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.829651 1267 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:32.829883 1918 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:32.830698 1267 tablet_bootstrap.cc:654] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.830957 1918 tablet_bootstrap.cc:654] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.831593 1920 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:32.831686 31971 tablet_service.cc:1467] Processing CreateTablet for tablet acdcc4d7e5754103a08d76bdd714a68e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.831881 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdcc4d7e5754103a08d76bdd714a68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.832434 1920 tablet_bootstrap.cc:654] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.833505 1918 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:32.833590 1918 ts_tablet_manager.cc:1367] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:00:32.834043 1918 raft_consensus.cc:348] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.834157 1918 raft_consensus.cc:374] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.834192 1918 raft_consensus.cc:729] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.834349 1918 consensus_queue.cc:260] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.834599 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2ab852e31d4181a8597b9047a08554 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.835191 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2ab852e31d4181a8597b9047a08554. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.835575 32235 tablet_service.cc:1467] Processing CreateTablet for tablet acdcc4d7e5754103a08d76bdd714a68e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.835799 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdcc4d7e5754103a08d76bdd714a68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.835945 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2ab852e31d4181a8597b9047a08554 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.836154 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2ab852e31d4181a8597b9047a08554. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.836961 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce0cb93decb4313a1677fd5613d1749 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.837250 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce0cb93decb4313a1677fd5613d1749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.837846 31840 tablet_service.cc:1467] Processing CreateTablet for tablet acdcc4d7e5754103a08d76bdd714a68e (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.838156 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdcc4d7e5754103a08d76bdd714a68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.838236 1918 ts_tablet_manager.cc:1398] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:32.838393 1918 tablet_bootstrap.cc:492] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:32.839154 1918 tablet_bootstrap.cc:654] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.839736 1700 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:32.840672 1700 tablet_bootstrap.cc:654] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.840808 1920 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:32.840891 1920 ts_tablet_manager.cc:1367] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:32.841325 1920 raft_consensus.cc:348] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:32.841463 1920 raft_consensus.cc:374] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.841503 1920 raft_consensus.cc:729] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.841737 1700 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:32.841825 1700 ts_tablet_manager.cc:1367] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:32.842274 1700 raft_consensus.cc:348] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:32.842394 1700 raft_consensus.cc:374] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.842362 1920 consensus_queue.cc:260] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:32.842434 1700 raft_consensus.cc:729] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.842571 1923 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:32.842561 1700 consensus_queue.cc:260] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:32.843170 1920 ts_tablet_manager.cc:1398] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:32.844367 1923 tablet_bootstrap.cc:654] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.843330 1920 tablet_bootstrap.cc:492] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:32.845526 1700 ts_tablet_manager.cc:1398] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:32.845613 1267 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:32.845700 1267 ts_tablet_manager.cc:1367] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240502 14:00:32.846195 1267 raft_consensus.cc:348] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.846314 1267 raft_consensus.cc:374] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.846321 1920 tablet_bootstrap.cc:654] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.846352 1267 raft_consensus.cc:729] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.846495 1267 consensus_queue.cc:260] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.846828 1267 ts_tablet_manager.cc:1398] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:32.847057 1267 tablet_bootstrap.cc:492] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:32.847589 1920 tablet_bootstrap.cc:492] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:32.847762 1920 ts_tablet_manager.cc:1367] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:32.848094 1267 tablet_bootstrap.cc:654] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.848280 1920 raft_consensus.cc:348] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.848410 1920 raft_consensus.cc:374] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.848445 1920 raft_consensus.cc:729] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.848609 1920 consensus_queue.cc:260] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.848889 1920 ts_tablet_manager.cc:1398] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:32.850677 1904 raft_consensus.cc:482] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.850781 1904 raft_consensus.cc:504] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.851209 1904 leader_election.cc:288] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:32.851613 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce0cb93decb4313a1677fd5613d1749" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:32.851728 31861 raft_consensus.cc:2454] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:32.852136 32188 leader_election.cc:302] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:32.852434 1904 raft_consensus.cc:2781] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.852491 1904 raft_consensus.cc:482] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.852531 1904 raft_consensus.cc:3037] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.852794 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce0cb93decb4313a1677fd5613d1749" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:32.853582 1904 raft_consensus.cc:504] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.853906 1904 leader_election.cc:288] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
W20240502 14:00:32.854406 32189 leader_election.cc:341] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:32.854982 1267 tablet_bootstrap.cc:492] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:32.855082 1267 ts_tablet_manager.cc:1367] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:00:32.855376 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce0cb93decb4313a1677fd5613d1749" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:32.855484 31861 raft_consensus.cc:3037] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.855670 1267 raft_consensus.cc:348] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.855787 1267 raft_consensus.cc:374] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.855825 1267 raft_consensus.cc:729] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.855962 1267 consensus_queue.cc:260] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.856326 1267 ts_tablet_manager.cc:1398] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.856467 31861 raft_consensus.cc:2454] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:32.856523 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce0cb93decb4313a1677fd5613d1749" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:32.856796 32188 leader_election.cc:302] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
W20240502 14:00:32.857510 32189 leader_election.cc:341] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:32.858135 1930 ts_tablet_manager.cc:902] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:32.858707 1930 tablet_copy_client.cc:308] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:32.860512 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 8e41bff5f1484f008851e76cc095eb91 from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:34141)
I20240502 14:00:32.860627 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 8e41bff5f1484f008851e76cc095eb91 from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:34141: session id = 8973f2527acb48f5ad36589652de73e1-8e41bff5f1484f008851e76cc095eb91
I20240502 14:00:32.861567 29884 tablet_copy_source_session.cc:205] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:32.862543 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e41bff5f1484f008851e76cc095eb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.863773 1923 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:32.863772 1918 tablet_bootstrap.cc:492] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:32.863878 1923 ts_tablet_manager.cc:1367] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240502 14:00:32.863878 1918 ts_tablet_manager.cc:1367] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240502 14:00:32.864393 1923 raft_consensus.cc:348] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.864410 1918 raft_consensus.cc:348] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.864517 1918 raft_consensus.cc:374] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.864518 1923 raft_consensus.cc:374] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.864553 1918 raft_consensus.cc:729] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.864553 1923 raft_consensus.cc:729] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.864691 1918 consensus_queue.cc:260] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.864691 1923 consensus_queue.cc:260] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.864989 1923 ts_tablet_manager.cc:1398] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.864988 1918 ts_tablet_manager.cc:1398] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.865183 1918 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:32.865219 1930 tablet_copy_client.cc:783] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:32.865362 1930 tablet_copy_client.cc:647] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:32.866197 1918 tablet_bootstrap.cc:654] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.867146 1306 raft_consensus.cc:2781] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.867336 1306 raft_consensus.cc:686] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:32.867465 1918 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:32.867535 1918 ts_tablet_manager.cc:1367] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:32.867484 1306 consensus_queue.cc:237] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.868342 1918 raft_consensus.cc:348] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:32.868443 1918 raft_consensus.cc:374] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.868484 1918 raft_consensus.cc:729] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.868609 1918 consensus_queue.cc:260] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:32.869112 1918 ts_tablet_manager.cc:1398] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:32.873128 1930 tablet_copy_client.cc:523] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:32.875510 1902 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:32.875882 1902 tablet_bootstrap.cc:492] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:32.876066 1902 ts_tablet_manager.cc:1367] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.052s user 0.023s sys 0.005s
I20240502 14:00:32.875932 29322 catalog_manager.cc:5484] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:32.876739 1902 raft_consensus.cc:348] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.876856 1902 raft_consensus.cc:729] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.876981 1902 consensus_queue.cc:260] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:32.877280 1902 ts_tablet_manager.cc:1398] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.877878 1930 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:32.878124 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-93e77c40b5674865ba6ccce626545ec7 received from {username='slave'} at 127.22.55.135:55687
I20240502 14:00:32.878214 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-93e77c40b5674865ba6ccce626545ec7 on tablet 93e77c40b5674865ba6ccce626545ec7 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:32.888017 1941 raft_consensus.cc:482] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.888139 1941 raft_consensus.cc:504] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.889187 1941 leader_election.cc:288] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:32.889293 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdcc4d7e5754103a08d76bdd714a68e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:32.889441 31861 raft_consensus.cc:2454] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:32.889602 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdcc4d7e5754103a08d76bdd714a68e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:32.889727 32188 leader_election.cc:302] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:32.889775 31990 raft_consensus.cc:2454] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:32.890143 1904 raft_consensus.cc:2781] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.890202 1904 raft_consensus.cc:482] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.890240 1904 raft_consensus.cc:3037] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.891295 1904 raft_consensus.cc:504] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.891593 1904 leader_election.cc:288] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:32.894305 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdcc4d7e5754103a08d76bdd714a68e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:32.894414 31990 raft_consensus.cc:3037] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.894428 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdcc4d7e5754103a08d76bdd714a68e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:32.894536 31861 raft_consensus.cc:3037] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.895430 31990 raft_consensus.cc:2454] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:32.895483 31861 raft_consensus.cc:2454] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:32.895969 32188 leader_election.cc:302] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:32.896381 1941 raft_consensus.cc:2781] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.896438 29328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:32.896565 29328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:32.896508 1941 raft_consensus.cc:686] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:32.896859 1941 consensus_queue.cc:237] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:32.901870 29322 catalog_manager.cc:5484] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:32.906710 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 1ffa686c8a8e47a5bf770e236b915b24
I20240502 14:00:32.907266 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffa686c8a8e47a5bf770e236b915b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:32.907593 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffa686c8a8e47a5bf770e236b915b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:32.908285 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffa686c8a8e47a5bf770e236b915b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:32 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:32.909243 1952 tablet_replica.cc:330] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:32.909440 1952 raft_consensus.cc:2229] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:32.909691 1952 raft_consensus.cc:2258] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.910341 1952 ts_tablet_manager.cc:1875] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.912218 1951 tablet_replica.cc:330] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:32.912477 1951 raft_consensus.cc:2229] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.912698 1951 raft_consensus.cc:2258] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.912704 1952 ts_tablet_manager.cc:1888] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:32.912771 1952 log.cc:1196] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/1ffa686c8a8e47a5bf770e236b915b24
I20240502 14:00:32.913146 1952 ts_tablet_manager.cc:1909] T 1ffa686c8a8e47a5bf770e236b915b24 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:32.913651 1951 ts_tablet_manager.cc:1875] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.913936 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 1ffa686c8a8e47a5bf770e236b915b24 (table rebalancer_dynamic_table_new_name [id=937cd3d02e71476c8c5c7081919f1fd8]) successfully deleted
I20240502 14:00:32.915128 1930 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:32.915418 1930 tablet_bootstrap.cc:492] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:32.915592 1930 ts_tablet_manager.cc:1367] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.003s
I20240502 14:00:32.916049 1930 raft_consensus.cc:348] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:32.916213 1930 raft_consensus.cc:729] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.916388 1951 ts_tablet_manager.cc:1888] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:32.916365 1930 consensus_queue.cc:260] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:32.916610 1951 log.cc:1196] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/1ffa686c8a8e47a5bf770e236b915b24
I20240502 14:00:32.916687 1930 ts_tablet_manager.cc:1398] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:32.917006 1951 ts_tablet_manager.cc:1909] T 1ffa686c8a8e47a5bf770e236b915b24 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:32.917433 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-8e41bff5f1484f008851e76cc095eb91 received from {username='slave'} at 127.22.55.136:34141
I20240502 14:00:32.917516 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-8e41bff5f1484f008851e76cc095eb91 on tablet 8e41bff5f1484f008851e76cc095eb91 with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:32.918219 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 1ffa686c8a8e47a5bf770e236b915b24 (table rebalancer_dynamic_table_new_name [id=937cd3d02e71476c8c5c7081919f1fd8]) successfully deleted
I20240502 14:00:32.919013 1953 tablet_replica.cc:330] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:32.919173 1953 raft_consensus.cc:2229] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:32.919313 1953 raft_consensus.cc:2258] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:32.919924 1953 ts_tablet_manager.cc:1875] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:32.922202 1953 ts_tablet_manager.cc:1888] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:32.922335 1953 log.cc:1196] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/1ffa686c8a8e47a5bf770e236b915b24
I20240502 14:00:32.922725 1953 ts_tablet_manager.cc:1909] T 1ffa686c8a8e47a5bf770e236b915b24 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:32.923537 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 1ffa686c8a8e47a5bf770e236b915b24 (table rebalancer_dynamic_table_new_name [id=937cd3d02e71476c8c5c7081919f1fd8]) successfully deleted
I20240502 14:00:32.954895 1983 ts_tablet_manager.cc:902] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:32.955478 1983 tablet_copy_client.cc:308] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:32.955926 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet aeaf1db0298741fa813660a8e38d01f9 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:34855)
I20240502 14:00:32.956032 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet aeaf1db0298741fa813660a8e38d01f9 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:34855: session id = f0e01c4f581843e9ad4ced97f0072dc4-aeaf1db0298741fa813660a8e38d01f9
I20240502 14:00:32.957068 29491 tablet_copy_source_session.cc:205] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:32.957950 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaf1db0298741fa813660a8e38d01f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.960120 1983 tablet_copy_client.cc:783] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:32.960278 1983 tablet_copy_client.cc:647] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:32.971592 1983 tablet_copy_client.cc:523] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:32.972571 1988 ts_tablet_manager.cc:902] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:32.972980 1983 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:32.977147 1988 tablet_copy_client.cc:308] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:00:32.977888 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet 27e9d8ef4f754df3817cc4a981549e02 from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:35643)
I20240502 14:00:32.978014 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet 27e9d8ef4f754df3817cc4a981549e02 from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:35643: session id = efdbf7bdfe0b41bdadb1add053b09a7e-27e9d8ef4f754df3817cc4a981549e02
I20240502 14:00:32.979214 31865 tablet_copy_source_session.cc:205] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:32.980693 1988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e9d8ef4f754df3817cc4a981549e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.981628 1080 raft_consensus.cc:482] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.981714 1080 raft_consensus.cc:504] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.982158 1080 leader_election.cc:288] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:32.982601 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce0cb93decb4313a1677fd5613d1749" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:32.982772 31861 raft_consensus.cc:2379] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate efdbf7bdfe0b41bdadb1add053b09a7e in current term 1: Already voted for candidate 8973f2527acb48f5ad36589652de73e1 in this term.
I20240502 14:00:32.983131 1988 tablet_copy_client.cc:783] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:32.983317 1988 tablet_copy_client.cc:647] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:32.986766 1988 tablet_copy_client.cc:523] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:32.988385 1988 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:32.988996 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce0cb93decb4313a1677fd5613d1749" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:32.989413 29668 leader_election.cc:302] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:32.989634 1080 raft_consensus.cc:3037] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.990561 1080 raft_consensus.cc:2726] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:00:32.992408 1988 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:32.992614 1988 tablet_bootstrap.cc:492] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:00:32.992733 1988 ts_tablet_manager.cc:1367] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:00:32.993139 1988 raft_consensus.cc:348] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.993301 1988 raft_consensus.cc:729] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.993456 1988 consensus_queue.cc:260] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:32.993799 1988 ts_tablet_manager.cc:1398] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:32.994741 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-27e9d8ef4f754df3817cc4a981549e02 received from {username='slave'} at 127.22.55.131:35643
I20240502 14:00:32.994840 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-27e9d8ef4f754df3817cc4a981549e02 on tablet 27e9d8ef4f754df3817cc4a981549e02 with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:32.999504 1983 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:32.999781 1983 tablet_bootstrap.cc:492] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:32.999956 1983 ts_tablet_manager.cc:1367] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.007s
I20240502 14:00:33.000344 1983 raft_consensus.cc:348] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.000452 1983 raft_consensus.cc:729] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.000581 1983 consensus_queue.cc:260] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.000846 1983 ts_tablet_manager.cc:1398] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:33.001364 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-aeaf1db0298741fa813660a8e38d01f9 received from {username='slave'} at 127.22.55.133:34855
I20240502 14:00:33.001457 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-aeaf1db0298741fa813660a8e38d01f9 on tablet aeaf1db0298741fa813660a8e38d01f9 with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:33.009265 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:33.014194 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 729d9f1482754446b50a861031f4df3d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f]), partition=
I20240502 14:00:33.014281 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 729d9f1482754446b50a861031f4df3d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f]), partition=
I20240502 14:00:33.014568 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729d9f1482754446b50a861031f4df3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.014664 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 729d9f1482754446b50a861031f4df3d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f]), partition=
I20240502 14:00:33.014899 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729d9f1482754446b50a861031f4df3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.016281 1918 tablet_bootstrap.cc:492] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:33.016556 1267 tablet_bootstrap.cc:492] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:33.016847 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729d9f1482754446b50a861031f4df3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.017102 1918 tablet_bootstrap.cc:654] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.017367 1267 tablet_bootstrap.cc:654] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.018107 1918 tablet_bootstrap.cc:492] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:33.018193 1918 ts_tablet_manager.cc:1367] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:33.018440 1267 tablet_bootstrap.cc:492] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:33.018507 1267 ts_tablet_manager.cc:1367] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:33.018632 1918 raft_consensus.cc:348] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.018750 1918 raft_consensus.cc:374] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.018782 1918 raft_consensus.cc:729] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.018925 1918 consensus_queue.cc:260] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.018982 1267 raft_consensus.cc:348] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.019084 1267 raft_consensus.cc:374] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.019129 1267 raft_consensus.cc:729] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.019161 1700 tablet_bootstrap.cc:492] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:33.019201 1918 ts_tablet_manager.cc:1398] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:33.019265 1267 consensus_queue.cc:260] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.019532 1267 ts_tablet_manager.cc:1398] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.020051 1700 tablet_bootstrap.cc:654] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.021075 1700 tablet_bootstrap.cc:492] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:33.021168 1700 ts_tablet_manager.cc:1367] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:33.021621 1700 raft_consensus.cc:348] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.021726 1700 raft_consensus.cc:374] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.021759 1700 raft_consensus.cc:729] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.021879 1700 consensus_queue.cc:260] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.022125 1700 ts_tablet_manager.cc:1398] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.029095 1825 raft_consensus.cc:482] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:33.029208 1825 raft_consensus.cc:504] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.029620 1825 leader_election.cc:288] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:33.029963 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2ab852e31d4181a8597b9047a08554" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:33.030004 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2ab852e31d4181a8597b9047a08554" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:33.030123 32254 raft_consensus.cc:2454] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:33.030123 31990 raft_consensus.cc:2454] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:33.030398 29537 leader_election.cc:302] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:33.030560 1825 raft_consensus.cc:2781] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:33.030616 1825 raft_consensus.cc:482] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:33.030642 1825 raft_consensus.cc:3037] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.031847 1825 raft_consensus.cc:504] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.032209 1825 leader_election.cc:288] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:33.032488 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2ab852e31d4181a8597b9047a08554" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:33.032564 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2ab852e31d4181a8597b9047a08554" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:33.032617 31990 raft_consensus.cc:3037] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.032682 32254 raft_consensus.cc:3037] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.033581 31990 raft_consensus.cc:2454] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:33.033581 32254 raft_consensus.cc:2454] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:33.033852 29535 leader_election.cc:302] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:33.034024 1825 raft_consensus.cc:2781] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:33.034109 1825 raft_consensus.cc:686] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:33.034291 1825 consensus_queue.cc:237] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.036331 1897 ts_tablet_manager.cc:902] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:33.036389 29323 catalog_manager.cc:5484] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:33.036798 1897 tablet_copy_client.cc:308] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:33.037729 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet fe523f0c364f447cb2a62b599d030cca from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:44929)
I20240502 14:00:33.037848 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet fe523f0c364f447cb2a62b599d030cca from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:44929: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-fe523f0c364f447cb2a62b599d030cca
I20240502 14:00:33.038954 32012 tablet_copy_source_session.cc:205] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:33.039988 1897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe523f0c364f447cb2a62b599d030cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.041749 1897 tablet_copy_client.cc:783] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:33.041904 1897 tablet_copy_client.cc:647] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:33.043300 1897 tablet_copy_client.cc:523] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:33.044493 1897 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:33.047533 858 raft_consensus.cc:482] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:33.047626 858 raft_consensus.cc:504] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.047818 1897 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:33.048020 858 leader_election.cc:288] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:33.048079 1897 tablet_bootstrap.cc:492] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:00:33.048255 1897 ts_tablet_manager.cc:1367] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 14:00:33.048408 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729d9f1482754446b50a861031f4df3d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:33.048440 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729d9f1482754446b50a861031f4df3d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:33.048538 32254 raft_consensus.cc:2454] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:33.048553 29601 raft_consensus.cc:2454] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:33.048760 1897 raft_consensus.cc:348] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:33.048848 31796 leader_election.cc:302] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:33.048899 1897 raft_consensus.cc:729] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.049075 1897 consensus_queue.cc:260] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:33.049209 858 raft_consensus.cc:2781] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:33.049271 858 raft_consensus.cc:482] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:33.049309 858 raft_consensus.cc:3037] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.050055 1897 ts_tablet_manager.cc:1398] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:33.050148 858 raft_consensus.cc:504] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.050510 858 leader_election.cc:288] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:33.050734 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-fe523f0c364f447cb2a62b599d030cca received from {username='slave'} at 127.22.55.132:44929
I20240502 14:00:33.050823 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-fe523f0c364f447cb2a62b599d030cca on tablet fe523f0c364f447cb2a62b599d030cca with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:33.050801 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729d9f1482754446b50a861031f4df3d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:33.050952 29601 raft_consensus.cc:3037] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.051908 29601 raft_consensus.cc:2454] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:33.052178 31794 leader_election.cc:302] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:33.052317 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729d9f1482754446b50a861031f4df3d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:33.052448 32254 raft_consensus.cc:3037] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.053387 32254 raft_consensus.cc:2454] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:33.053856 858 raft_consensus.cc:2781] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:33.053956 858 raft_consensus.cc:686] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:33.054108 858 consensus_queue.cc:237] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:33.056367 29323 catalog_manager.cc:5484] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:33.066498 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:33.068945 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.069057 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.076435 32233 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.088649 29582 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.096691 32101 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.097535 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.099777 31840 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.138339 1902 ts_tablet_manager.cc:902] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:33.138818 1902 tablet_copy_client.cc:308] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:33.139283 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 9790e7ee6d8945bf85f129ef3f9900ca from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37167)
I20240502 14:00:33.139405 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 9790e7ee6d8945bf85f129ef3f9900ca from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37167: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-9790e7ee6d8945bf85f129ef3f9900ca
I20240502 14:00:33.140413 29884 tablet_copy_source_session.cc:205] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:33.141542 1902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9790e7ee6d8945bf85f129ef3f9900ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.143872 1902 tablet_copy_client.cc:783] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:33.144027 1902 tablet_copy_client.cc:647] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:33.146378 1902 tablet_copy_client.cc:523] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:33.147631 1902 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:33.151381 1902 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:33.151619 1902 tablet_bootstrap.cc:492] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:33.151779 1902 ts_tablet_manager.cc:1367] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 14:00:33.152184 1902 raft_consensus.cc:348] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.152302 1902 raft_consensus.cc:729] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.152484 1902 consensus_queue.cc:260] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.152750 1902 ts_tablet_manager.cc:1398] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:00:33.153295 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-9790e7ee6d8945bf85f129ef3f9900ca received from {username='slave'} at 127.22.55.135:37167
I20240502 14:00:33.153388 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-9790e7ee6d8945bf85f129ef3f9900ca on tablet 9790e7ee6d8945bf85f129ef3f9900ca with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:33.172612 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:33.177178 32254 raft_consensus.cc:1261] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.177167 29601 raft_consensus.cc:1261] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.177610 1075 consensus_queue.cc:1035] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.177942 858 consensus_queue.cc:1035] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:00:33.181244 29323 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 729d9f1482754446b50a861031f4df3d (table rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f]). Expected version 1 got 0
I20240502 14:00:33.182516 29323 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 729d9f1482754446b50a861031f4df3d (table rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f]). Expected version 1 got 0
I20240502 14:00:33.183352 1373 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:33.183663 1373 tablet.cc:1620] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:33.184667 2045 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:33.184731 1770 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:33.184988 2045 tablet.cc:1620] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:33.185106 1770 tablet.cc:1620] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:33.188692 29325 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 729d9f1482754446b50a861031f4df3d (table rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f]). Expected version 1 got 0
I20240502 14:00:33.190225 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f] alter complete (version 1)
I20240502 14:00:33.223457 2075 ts_tablet_manager.cc:902] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:33.224938 2075 tablet_copy_client.cc:308] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:33.225646 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 70367c7c718d4d9195f0af7e69bdf2ec from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:32873)
I20240502 14:00:33.225747 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 70367c7c718d4d9195f0af7e69bdf2ec from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:32873: session id = 1d00d3a47c52478894cf8c66959ef7bd-70367c7c718d4d9195f0af7e69bdf2ec
I20240502 14:00:33.226799 32143 tablet_copy_source_session.cc:205] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:33.228238 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70367c7c718d4d9195f0af7e69bdf2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.228324 1897 ts_tablet_manager.cc:902] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:33.228750 1897 tablet_copy_client.cc:308] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:33.229225 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet e9f48a525d8c4a0783f0450aa6ec7cb0 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:53217)
I20240502 14:00:33.229326 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet e9f48a525d8c4a0783f0450aa6ec7cb0 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:53217: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-e9f48a525d8c4a0783f0450aa6ec7cb0
I20240502 14:00:33.230242 29622 tablet_copy_source_session.cc:205] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:33.232223 1897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f48a525d8c4a0783f0450aa6ec7cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.234405 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.235567 2075 tablet_copy_client.cc:783] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:33.235756 2075 tablet_copy_client.cc:647] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:33.234417 1897 tablet_copy_client.cc:783] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:33.237867 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.237989 1897 tablet_copy_client.cc:647] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:33.239456 1897 tablet_copy_client.cc:523] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:33.240854 1897 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:33.244238 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:33.246179 29863 raft_consensus.cc:1203] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:33.246544 1897 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:33.246795 1897 tablet_bootstrap.cc:492] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:00:33.246987 1897 ts_tablet_manager.cc:1367] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:00:33.247352 1897 raft_consensus.cc:348] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:33.247435 1897 raft_consensus.cc:729] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.247524 1897 consensus_queue.cc:260] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:33.247759 1897 ts_tablet_manager.cc:1398] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:33.248278 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-e9f48a525d8c4a0783f0450aa6ec7cb0 received from {username='slave'} at 127.22.55.132:53217
I20240502 14:00:33.248359 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-e9f48a525d8c4a0783f0450aa6ec7cb0 on tablet e9f48a525d8c4a0783f0450aa6ec7cb0 with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:33.251158 32233 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.251756 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.252353 32119 raft_consensus.cc:1203] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:33.255195 32254 raft_consensus.cc:1203] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:33.264024 29582 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.269203 31840 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.269699 32101 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.281327 2075 tablet_copy_client.cc:523] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:33.291247 2075 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:33.295634 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:33.301223 1373 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:33.301586 1373 tablet.cc:1620] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:33.301590 1770 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:33.301767 2045 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:33.301851 1770 tablet.cc:1620] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:33.302037 2045 tablet.cc:1620] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:33.308033 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=658fcbf30a6c4cf987d9bfc75c5c712f] alter complete (version 2)
I20240502 14:00:33.314036 2075 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:33.314461 2075 tablet_bootstrap.cc:492] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:00:33.314656 2075 ts_tablet_manager.cc:1367] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20240502 14:00:33.315234 2075 raft_consensus.cc:348] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:33.315348 2075 raft_consensus.cc:729] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.315515 2075 consensus_queue.cc:260] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:33.315896 2075 ts_tablet_manager.cc:1398] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.316553 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-70367c7c718d4d9195f0af7e69bdf2ec received from {username='slave'} at 127.22.55.129:32873
I20240502 14:00:33.316643 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-70367c7c718d4d9195f0af7e69bdf2ec on tablet 70367c7c718d4d9195f0af7e69bdf2ec with peer 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:33.379655 1941 consensus_queue.cc:1035] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.385337 1904 consensus_queue.cc:1035] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.410384 29324 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:33.418097 1373 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:33.419155 1770 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:33.421124 1941 consensus_queue.cc:1035] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.429132 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=658fcbf30a6c4cf987d9bfc75c5c712f] alter complete (version 3)
I20240502 14:00:33.430146 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.430745 2045 tablet.cc:1722] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:33.432186 29582 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.435133 31861 raft_consensus.cc:1203] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:33.436926 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.438369 1904 consensus_queue.cc:1035] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.439863 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:33.442422 32101 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.450222 32233 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.453410 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.464239 31840 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.490769 1825 consensus_queue.cc:1035] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.494295 1825 consensus_queue.cc:1035] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.520627 29863 raft_consensus.cc:1203] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:33.531090 29863 consensus_queue.cc:237] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.532490 32254 raft_consensus.cc:1261] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:33.532624 29601 raft_consensus.cc:1261] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:33.532881 32672 consensus_queue.cc:1035] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:33.533167 32672 consensus_queue.cc:1035] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:33.535077 32672 raft_consensus.cc:2932] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:33.535660 29601 raft_consensus.cc:2932] T 8e41bff5f1484f008851e76cc095eb91 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:33.536789 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:33.536922 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:33.538925 29324 catalog_manager.cc:5484] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:33.542133 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e41bff5f1484f008851e76cc095eb91 with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:33.542378 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 729d9f1482754446b50a861031f4df3d
I20240502 14:00:33.542924 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 729d9f1482754446b50a861031f4df3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:33 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:33.542959 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 729d9f1482754446b50a861031f4df3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:33 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:33.543186 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 729d9f1482754446b50a861031f4df3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:33 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:33.543232 2175 tablet_replica.cc:330] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:33.543373 2175 raft_consensus.cc:2229] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:33.543601 2175 raft_consensus.cc:2258] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:33.544026 2174 tablet_replica.cc:330] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:33.544188 2174 raft_consensus.cc:2229] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:33.544749 2176 tablet_replica.cc:330] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:33.544817 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:33.545130 2176 raft_consensus.cc:2229] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:33.545197 2175 ts_tablet_manager.cc:1875] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:33.545459 2176 raft_consensus.cc:2258] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:33.545562 2174 raft_consensus.cc:2258] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:33.548462 2175 ts_tablet_manager.cc:1888] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:33.548583 2175 log.cc:1196] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/729d9f1482754446b50a861031f4df3d
I20240502 14:00:33.548899 2175 ts_tablet_manager.cc:1909] T 729d9f1482754446b50a861031f4df3d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:33.549031 2176 ts_tablet_manager.cc:1875] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:33.549165 2174 ts_tablet_manager.cc:1875] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:33.549559 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 729d9f1482754446b50a861031f4df3d (table rebalancer_dynamic_table_new_name [id=658fcbf30a6c4cf987d9bfc75c5c712f]) successfully deleted
I20240502 14:00:33.549996 2173 tablet_replica.cc:330] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:33.550146 2173 raft_consensus.cc:2229] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:33.550300 2173 raft_consensus.cc:2258] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:33.550829 31840 tablet_service.cc:1467] Processing CreateTablet for tablet fbdb9b4a6c07488f9bc2d15cba59214e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.550900 2173 ts_tablet_manager.cc:1875] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:33.551124 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdb9b4a6c07488f9bc2d15cba59214e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.551245 32101 tablet_service.cc:1467] Processing CreateTablet for tablet fbdb9b4a6c07488f9bc2d15cba59214e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.551334 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea3465ad91540cea8906bcc406efd07 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.551471 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdb9b4a6c07488f9bc2d15cba59214e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.551570 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea3465ad91540cea8906bcc406efd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.551903 31971 tablet_service.cc:1467] Processing CreateTablet for tablet fbdb9b4a6c07488f9bc2d15cba59214e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.552145 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdb9b4a6c07488f9bc2d15cba59214e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.553478 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea3465ad91540cea8906bcc406efd07 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.553702 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea3465ad91540cea8906bcc406efd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.554309 2184 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:33.555251 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 5733940f27f345d8a314a9e51d2a455e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.555471 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5733940f27f345d8a314a9e51d2a455e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.555471 2184 tablet_bootstrap.cc:654] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.555774 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 5733940f27f345d8a314a9e51d2a455e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.555999 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5733940f27f345d8a314a9e51d2a455e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.550061 32254 raft_consensus.cc:2932] T 8e41bff5f1484f008851e76cc095eb91 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:33.554029 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea3465ad91540cea8906bcc406efd07 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.557796 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea3465ad91540cea8906bcc406efd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.557758 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 5733940f27f345d8a314a9e51d2a455e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:33.557951 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5733940f27f345d8a314a9e51d2a455e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.559947 2186 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:33.561050 2186 tablet_bootstrap.cc:654] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.563592 2174 ts_tablet_manager.cc:1888] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:33.563717 2174 log.cc:1196] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/729d9f1482754446b50a861031f4df3d
I20240502 14:00:33.564044 2174 ts_tablet_manager.cc:1909] T 729d9f1482754446b50a861031f4df3d P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:33.564103 2173 ts_tablet_manager.cc:1888] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:33.564189 2173 log.cc:1196] T 8e41bff5f1484f008851e76cc095eb91 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/8e41bff5f1484f008851e76cc095eb91
I20240502 14:00:33.564204 2184 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:33.564291 2184 ts_tablet_manager.cc:1367] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:33.564777 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 8e41bff5f1484f008851e76cc095eb91 (table rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]) successfully deleted
I20240502 14:00:33.564790 2184 raft_consensus.cc:348] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.564909 2184 raft_consensus.cc:374] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.564951 2184 raft_consensus.cc:729] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.565083 2184 consensus_queue.cc:260] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.565378 2184 ts_tablet_manager.cc:1398] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.565481 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 729d9f1482754446b50a861031f4df3d (table rebalancer_dynamic_table_new_name [id=658fcbf30a6c4cf987d9bfc75c5c712f]) successfully deleted
I20240502 14:00:33.565565 2184 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:33.566298 2183 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:33.566416 2184 tablet_bootstrap.cc:654] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.567260 2183 tablet_bootstrap.cc:654] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.567992 2176 ts_tablet_manager.cc:1888] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:33.568154 2176 log.cc:1196] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/729d9f1482754446b50a861031f4df3d
I20240502 14:00:33.568519 2176 ts_tablet_manager.cc:1909] T 729d9f1482754446b50a861031f4df3d P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:33.569264 2184 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:33.569262 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 729d9f1482754446b50a861031f4df3d (table rebalancer_dynamic_table_new_name [id=658fcbf30a6c4cf987d9bfc75c5c712f]) successfully deleted
I20240502 14:00:33.569342 2182 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:33.569344 2184 ts_tablet_manager.cc:1367] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:33.569756 2184 raft_consensus.cc:348] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.569895 2184 raft_consensus.cc:374] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.569936 2184 raft_consensus.cc:729] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.570060 2184 consensus_queue.cc:260] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.570657 2184 ts_tablet_manager.cc:1398] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.570780 2182 tablet_bootstrap.cc:654] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.570822 2184 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:33.572213 2186 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:33.572281 2184 tablet_bootstrap.cc:654] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.572317 2186 ts_tablet_manager.cc:1367] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:33.572775 2186 raft_consensus.cc:348] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.572892 2186 raft_consensus.cc:374] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.572930 2186 raft_consensus.cc:729] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.573033 2183 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:33.573047 2186 consensus_queue.cc:260] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.573137 2183 ts_tablet_manager.cc:1367] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:33.573328 2186 ts_tablet_manager.cc:1398] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:33.573534 2186 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:33.573596 2183 raft_consensus.cc:348] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.573702 2183 raft_consensus.cc:374] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.573740 2183 raft_consensus.cc:729] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.573843 2183 consensus_queue.cc:260] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.574056 2183 ts_tablet_manager.cc:1398] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:33.574187 2183 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:33.574424 2186 tablet_bootstrap.cc:654] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.574983 2183 tablet_bootstrap.cc:654] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.575452 2184 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:33.575521 2184 ts_tablet_manager.cc:1367] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:00:33.575938 2184 raft_consensus.cc:348] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.576033 2184 raft_consensus.cc:374] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.576068 2184 raft_consensus.cc:729] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.576174 2184 consensus_queue.cc:260] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.576478 2184 ts_tablet_manager.cc:1398] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:33.577075 2183 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:33.577162 2183 ts_tablet_manager.cc:1367] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:33.577512 2183 raft_consensus.cc:348] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.577600 2183 raft_consensus.cc:374] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.577627 2183 raft_consensus.cc:729] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.577723 2183 consensus_queue.cc:260] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.577965 2183 ts_tablet_manager.cc:1398] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:33.578086 2183 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:33.578338 2182 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:33.578421 2186 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:33.578424 2182 ts_tablet_manager.cc:1367] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:33.578485 2186 ts_tablet_manager.cc:1367] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:33.578930 2183 tablet_bootstrap.cc:654] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.578877 2186 raft_consensus.cc:348] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.578903 2182 raft_consensus.cc:348] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.578979 2186 raft_consensus.cc:374] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.578994 2182 raft_consensus.cc:374] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.579015 2186 raft_consensus.cc:729] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.579030 2182 raft_consensus.cc:729] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.579144 2186 consensus_queue.cc:260] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.579181 2182 consensus_queue.cc:260] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.579504 2186 ts_tablet_manager.cc:1398] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:33.580554 2183 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:33.581687 2183 ts_tablet_manager.cc:1367] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:33.582087 2183 raft_consensus.cc:348] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.582281 2183 raft_consensus.cc:374] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.582300 2182 ts_tablet_manager.cc:1398] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:33.582351 2183 raft_consensus.cc:729] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.582489 2183 consensus_queue.cc:260] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.582837 2183 ts_tablet_manager.cc:1398] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:33.583887 29729 raft_consensus.cc:1203] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:33.585086 1904 raft_consensus.cc:482] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:33.585225 1904 raft_consensus.cc:504] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.585594 1904 leader_election.cc:288] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:33.585988 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3465ad91540cea8906bcc406efd07" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:33.586138 32119 raft_consensus.cc:2454] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:33.586521 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3465ad91540cea8906bcc406efd07" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:33.586655 31990 raft_consensus.cc:2454] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:33.586920 32188 leader_election.cc:302] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:33.587100 1904 raft_consensus.cc:2781] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:33.587157 1904 raft_consensus.cc:482] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:33.587193 1904 raft_consensus.cc:3037] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.588224 1904 raft_consensus.cc:504] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.588559 1904 leader_election.cc:288] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:33.588874 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3465ad91540cea8906bcc406efd07" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:33.588938 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3465ad91540cea8906bcc406efd07" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:33.588975 32119 raft_consensus.cc:3037] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.589041 31990 raft_consensus.cc:3037] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.589923 32119 raft_consensus.cc:2454] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:33.589936 31990 raft_consensus.cc:2454] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:33.590205 32188 leader_election.cc:302] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:33.590886 1904 raft_consensus.cc:2781] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:33.590981 1904 raft_consensus.cc:686] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:33.591115 1904 consensus_queue.cc:237] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.592630 29327 catalog_manager.cc:5484] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:33.644688 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:33.646786 29471 raft_consensus.cc:1203] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:33.650622 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 02dff3ad4ba84230b5ac52e274310748 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945]), partition=
I20240502 14:00:33.650933 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dff3ad4ba84230b5ac52e274310748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.651111 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 02dff3ad4ba84230b5ac52e274310748 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945]), partition=
I20240502 14:00:33.651360 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dff3ad4ba84230b5ac52e274310748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.652658 2186 tablet_bootstrap.cc:492] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:33.653414 2226 tablet_bootstrap.cc:492] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:33.653472 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 02dff3ad4ba84230b5ac52e274310748 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945]), partition=
I20240502 14:00:33.653640 2186 tablet_bootstrap.cc:654] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.653710 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02dff3ad4ba84230b5ac52e274310748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:33.654417 2226 tablet_bootstrap.cc:654] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.655190 2186 tablet_bootstrap.cc:492] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:33.655283 2186 ts_tablet_manager.cc:1367] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:33.655766 2186 raft_consensus.cc:348] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.655848 2183 tablet_bootstrap.cc:492] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:33.655890 2186 raft_consensus.cc:374] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.655922 2186 raft_consensus.cc:729] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.656044 2186 consensus_queue.cc:260] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.656325 2186 ts_tablet_manager.cc:1398] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.656687 2183 tablet_bootstrap.cc:654] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:33.657027 2226 tablet_bootstrap.cc:492] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:33.657138 2226 ts_tablet_manager.cc:1367] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:33.657493 2226 raft_consensus.cc:348] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.657586 2226 raft_consensus.cc:374] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.657619 2226 raft_consensus.cc:729] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.657749 2226 consensus_queue.cc:260] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.658061 2226 ts_tablet_manager.cc:1398] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.659070 2183 tablet_bootstrap.cc:492] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:33.659175 2183 ts_tablet_manager.cc:1367] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:33.659623 2183 raft_consensus.cc:348] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.659746 2183 raft_consensus.cc:374] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:33.659795 2183 raft_consensus.cc:729] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:33.659916 2183 consensus_queue.cc:260] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.660183 2183 ts_tablet_manager.cc:1398] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:33.692812 32119 raft_consensus.cc:1203] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:33.697479 32752 raft_consensus.cc:482] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:33.697592 32752 raft_consensus.cc:504] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.697965 32752 leader_election.cc:288] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:33.698361 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:33.698400 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:33.698516 32254 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:33.698529 31990 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:33.698807 32056 leader_election.cc:302] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:33.698995 858 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:33.699091 858 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.699441 858 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:33.699563 32752 raft_consensus.cc:2781] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:33.699621 32752 raft_consensus.cc:482] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:33.699661 32752 raft_consensus.cc:3037] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.699755 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:33.699882 31990 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:33.700256 31794 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:33.700656 32752 raft_consensus.cc:504] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.700982 32752 leader_election.cc:288] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:33.701254 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:33.701258 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:33.701295 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:33.701354 32119 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:33.701383 31990 raft_consensus.cc:3037] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.701400 32254 raft_consensus.cc:3037] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.701541 858 raft_consensus.cc:2781] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:33.701597 858 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:33.701629 858 raft_consensus.cc:3037] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.702332 31990 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:33.702332 32254 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:33.702569 858 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.702625 32058 leader_election.cc:302] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:33.702874 858 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:33.703136 32752 raft_consensus.cc:2781] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:33.703310 32752 raft_consensus.cc:686] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:33.703290 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:33.703475 32119 raft_consensus.cc:3037] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.703562 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:33.703649 31990 raft_consensus.cc:3037] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.704404 32119 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:33.704447 31990 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:33.704718 31794 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:33.704885 858 raft_consensus.cc:2781] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:33.704972 858 raft_consensus.cc:686] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:33.703480 32752 consensus_queue.cc:237] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:33.705085 858 consensus_queue.cc:237] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:33.709707 29325 catalog_manager.cc:5484] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:33.709872 29322 catalog_manager.cc:5484] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:33.723702 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.741950 29582 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.743263 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.747853 32233 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:33.749730 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:33.749958 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.755435 31840 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.773453 1825 raft_consensus.cc:482] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:33.773576 1825 raft_consensus.cc:504] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.773957 1825 leader_election.cc:288] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:33.774749 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dff3ad4ba84230b5ac52e274310748" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:33.774906 32254 raft_consensus.cc:2454] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:33.775523 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dff3ad4ba84230b5ac52e274310748" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:33.775624 29537 leader_election.cc:302] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:33.775650 32119 raft_consensus.cc:2454] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:33.776024 1825 raft_consensus.cc:2781] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:33.776080 1825 raft_consensus.cc:482] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:33.776125 1825 raft_consensus.cc:3037] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.777189 1825 raft_consensus.cc:504] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.777518 1825 leader_election.cc:288] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:33.778890 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dff3ad4ba84230b5ac52e274310748" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:33.779031 32254 raft_consensus.cc:3037] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.780061 32254 raft_consensus.cc:2454] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:33.780529 29537 leader_election.cc:302] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:33.780827 1537 raft_consensus.cc:2781] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:33.780891 1537 raft_consensus.cc:686] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:33.781016 1537 consensus_queue.cc:237] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:33.783134 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02dff3ad4ba84230b5ac52e274310748" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:33.783254 32119 raft_consensus.cc:3037] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:33.783649 29326 catalog_manager.cc:5484] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:33.784214 32119 raft_consensus.cc:2454] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:33.789639 32103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:33.814165 31990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca"
dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45712
I20240502 14:00:33.814361 31990 raft_consensus.cc:595] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:33.842065 29863 raft_consensus.cc:1203] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:33.887217 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.890774 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:33.900135 32233 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:33.900172 31840 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:33.905728 29582 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:33.907866 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:33.911571 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:33.915030 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:33.927883 32119 raft_consensus.cc:1261] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.928412 2299 consensus_queue.cc:1035] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.928735 1721 raft_consensus.cc:979] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: : Instructing follower f0e01c4f581843e9ad4ced97f0072dc4 to start an election
I20240502 14:00:33.928838 1721 raft_consensus.cc:1067] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Signalling peer f0e01c4f581843e9ad4ced97f0072dc4 to start an election
I20240502 14:00:33.929229 32254 raft_consensus.cc:1261] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.929636 1825 consensus_queue.cc:1035] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.929836 31860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca"
dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
from {username='slave'} at 127.22.55.134:44335
I20240502 14:00:33.930011 31860 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:33.930080 31860 raft_consensus.cc:3037] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:33.931152 31860 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:33.931623 31860 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:33.932183 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:33.932301 32122 raft_consensus.cc:3037] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:33.932507 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:33.932606 29863 raft_consensus.cc:3037] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:33.932960 29328 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 02dff3ad4ba84230b5ac52e274310748 (table rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945]). Expected version 1 got 0
I20240502 14:00:33.933306 32122 raft_consensus.cc:2454] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 2.
I20240502 14:00:33.933332 29863 raft_consensus.cc:2454] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 2.
I20240502 14:00:33.934099 31794 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:33.934289 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe523f0c364f447cb2a62b599d030cca" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:33.934389 31990 raft_consensus.cc:3032] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:33.934417 31990 raft_consensus.cc:729] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:33.934545 31990 consensus_queue.cc:260] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:33.934723 31990 raft_consensus.cc:3037] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:33.935701 31990 raft_consensus.cc:2454] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 2.
I20240502 14:00:33.935917 858 raft_consensus.cc:2781] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:33.936004 858 raft_consensus.cc:686] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:33.936141 858 consensus_queue.cc:237] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:33.940634 29329 catalog_manager.cc:5484] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 1 to 2, leader changed from 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 2 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:33.949548 29328 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 02dff3ad4ba84230b5ac52e274310748 (table rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945]). Expected version 1 got 0
I20240502 14:00:33.954641 29323 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 02dff3ad4ba84230b5ac52e274310748 (table rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945]). Expected version 1 got 0
I20240502 14:00:33.952916 2306 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:33.956178 32103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:33.956679 2306 tablet.cc:1620] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:33.962296 29323 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 02dff3ad4ba84230b5ac52e274310748 (table rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945]). Expected version 1 got 0
I20240502 14:00:33.963636 2301 mvcc.cc:205] Tried to move back new op lower bound from 7023240945322831872 to 7023240944768360448. Current Snapshot: MvccSnapshot[applied={T|T < 7023240945322831872}]
I20240502 14:00:33.963992 2307 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:33.964325 2307 tablet.cc:1620] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:33.968724 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945] alter complete (version 1)
I20240502 14:00:33.968925 2305 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:33.969237 2305 tablet.cc:1620] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:33.999219 1904 consensus_queue.cc:1035] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.002907 1904 consensus_queue.cc:1035] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.073956 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:34.082132 2305 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:34.082502 2305 tablet.cc:1620] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:34.082491 2306 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:34.082760 2306 tablet.cc:1620] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:34.084043 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:34.088014 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.095017 29712 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.090865 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=eb4d218daa7d44f4955be65395b83945] alter complete (version 2)
I20240502 14:00:34.099256 2307 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:34.099586 2307 tablet.cc:1620] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:34.100309 32233 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:34.101636 29582 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:34.106192 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:34.107618 32103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:34.114981 31840 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:34.178586 29732 consensus_queue.cc:237] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:34.179839 32119 raft_consensus.cc:1261] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:34.179917 29600 raft_consensus.cc:1261] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:34.180238 2379 consensus_queue.cc:1035] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:34.180431 1080 consensus_queue.cc:1035] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:34.182323 2382 raft_consensus.cc:2932] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:34.182637 32119 raft_consensus.cc:2932] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:34.182737 29600 raft_consensus.cc:2932] T 93e77c40b5674865ba6ccce626545ec7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:34.185520 29320 catalog_manager.cc:5484] T 93e77c40b5674865ba6ccce626545ec7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:34.188969 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e77c40b5674865ba6ccce626545ec7 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:34.189378 2383 tablet_replica.cc:330] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:34.189529 2383 raft_consensus.cc:2229] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:34.189688 2383 raft_consensus.cc:2258] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.190457 2383 ts_tablet_manager.cc:1875] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:34.192777 2383 ts_tablet_manager.cc:1888] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:34.192852 2383 log.cc:1196] T 93e77c40b5674865ba6ccce626545ec7 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/93e77c40b5674865ba6ccce626545ec7
I20240502 14:00:34.193432 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 93e77c40b5674865ba6ccce626545ec7 (table rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]) successfully deleted
I20240502 14:00:34.200520 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:34.207393 2305 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:34.207860 2307 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:34.209966 2306 tablet.cc:1722] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:34.213668 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=eb4d218daa7d44f4955be65395b83945] alter complete (version 3)
I20240502 14:00:34.217175 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:34.223759 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae0eeb814e34729b6284d838d458ddf (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.224125 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae0eeb814e34729b6284d838d458ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.224610 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 21a2c1b2947c47289ace8cf41f2fbb4d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.224749 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae0eeb814e34729b6284d838d458ddf (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.224867 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a2c1b2947c47289ace8cf41f2fbb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.224749 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 54f01d68468f4afca23ca423748ddb58 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.225090 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae0eeb814e34729b6284d838d458ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.225237 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f01d68468f4afca23ca423748ddb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.226805 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 21a2c1b2947c47289ace8cf41f2fbb4d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.227108 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a2c1b2947c47289ace8cf41f2fbb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.227131 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 54f01d68468f4afca23ca423748ddb58 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.227327 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f01d68468f4afca23ca423748ddb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.227758 2396 tablet_bootstrap.cc:492] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:34.228652 2396 tablet_bootstrap.cc:654] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.230005 2396 tablet_bootstrap.cc:492] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:34.229998 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 54f01d68468f4afca23ca423748ddb58 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.230095 2396 ts_tablet_manager.cc:1367] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:34.230252 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f01d68468f4afca23ca423748ddb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.230487 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae0eeb814e34729b6284d838d458ddf (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.230753 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae0eeb814e34729b6284d838d458ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.230762 2395 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:34.230571 2396 raft_consensus.cc:348] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.231000 2396 raft_consensus.cc:374] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.231050 2396 raft_consensus.cc:729] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.231309 2396 consensus_queue.cc:260] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.232174 2395 tablet_bootstrap.cc:654] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.233289 2399 tablet_bootstrap.cc:492] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:34.233258 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 21a2c1b2947c47289ace8cf41f2fbb4d (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:34.233511 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a2c1b2947c47289ace8cf41f2fbb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.234336 2396 ts_tablet_manager.cc:1398] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:34.234985 2399 tablet_bootstrap.cc:654] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.236157 2399 tablet_bootstrap.cc:492] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:34.236284 2399 ts_tablet_manager.cc:1367] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:34.236544 2396 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:34.237114 2399 raft_consensus.cc:348] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.237217 2399 raft_consensus.cc:374] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.237252 2399 raft_consensus.cc:729] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.237421 2399 consensus_queue.cc:260] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.237531 2406 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:34.237767 2399 ts_tablet_manager.cc:1398] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:34.238462 2395 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:34.238556 2395 ts_tablet_manager.cc:1367] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:34.239251 2406 tablet_bootstrap.cc:654] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.239662 2395 raft_consensus.cc:348] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.239849 2395 raft_consensus.cc:374] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.239936 2395 raft_consensus.cc:729] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.240135 2395 consensus_queue.cc:260] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.240835 2396 tablet_bootstrap.cc:654] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.240872 2406 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:34.240950 2406 ts_tablet_manager.cc:1367] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:34.241439 2406 raft_consensus.cc:348] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.241494 1075 consensus_queue.cc:1035] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.241572 2406 raft_consensus.cc:374] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.241611 2406 raft_consensus.cc:729] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.241747 2406 consensus_queue.cc:260] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.241833 2398 tablet_bootstrap.cc:492] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:34.242041 2406 ts_tablet_manager.cc:1398] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:34.242221 2406 tablet_bootstrap.cc:492] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:34.243084 2406 tablet_bootstrap.cc:654] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.243089 2398 tablet_bootstrap.cc:654] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.246434 2396 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:34.246501 2395 ts_tablet_manager.cc:1398] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:00:34.246546 2396 ts_tablet_manager.cc:1367] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:34.246706 2395 tablet_bootstrap.cc:492] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:34.246948 2406 tablet_bootstrap.cc:492] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:34.247084 2406 ts_tablet_manager.cc:1367] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:34.247228 2396 raft_consensus.cc:348] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.247421 2396 raft_consensus.cc:374] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.247501 2396 raft_consensus.cc:729] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.248039 2395 tablet_bootstrap.cc:654] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.247893 2396 consensus_queue.cc:260] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.248390 2406 raft_consensus.cc:348] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.248551 2406 raft_consensus.cc:374] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.248601 2406 raft_consensus.cc:729] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.248797 2406 consensus_queue.cc:260] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.249184 2406 ts_tablet_manager.cc:1398] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:34.250481 2396 ts_tablet_manager.cc:1398] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:34.250708 2395 tablet_bootstrap.cc:492] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:34.250810 2395 ts_tablet_manager.cc:1367] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:34.251379 2395 raft_consensus.cc:348] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.251531 2395 raft_consensus.cc:374] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.251595 2395 raft_consensus.cc:729] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.251780 2395 consensus_queue.cc:260] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.251915 858 consensus_queue.cc:1035] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.252704 2395 ts_tablet_manager.cc:1398] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:34.258055 2398 tablet_bootstrap.cc:492] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:34.258173 2398 ts_tablet_manager.cc:1367] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:00:34.258636 2398 raft_consensus.cc:348] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.258741 2398 raft_consensus.cc:374] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.258774 2398 raft_consensus.cc:729] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.258947 2398 consensus_queue.cc:260] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.259230 2398 ts_tablet_manager.cc:1398] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:34.259400 2398 tablet_bootstrap.cc:492] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:34.260521 2398 tablet_bootstrap.cc:654] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.261691 2398 tablet_bootstrap.cc:492] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:34.261776 2398 ts_tablet_manager.cc:1367] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:34.262217 2398 raft_consensus.cc:348] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.262316 2398 raft_consensus.cc:374] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.262349 2398 raft_consensus.cc:729] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.262471 2398 consensus_queue.cc:260] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.262770 2398 ts_tablet_manager.cc:1398] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:34.294503 1825 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:34.294629 1825 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.295150 1825 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:34.295567 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a2c1b2947c47289ace8cf41f2fbb4d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:34.295727 32119 raft_consensus.cc:2454] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:34.296270 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a2c1b2947c47289ace8cf41f2fbb4d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:34.296398 29732 raft_consensus.cc:2454] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:34.296808 29535 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:34.296896 32752 consensus_queue.cc:1035] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.297061 1825 raft_consensus.cc:2781] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:34.297112 1825 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:34.297150 1825 raft_consensus.cc:3037] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.298305 1825 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.298727 1825 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:34.299785 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a2c1b2947c47289ace8cf41f2fbb4d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:34.299899 29732 raft_consensus.cc:3037] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.300880 29732 raft_consensus.cc:2454] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:34.301167 29536 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:34.301172 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a2c1b2947c47289ace8cf41f2fbb4d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:34.301277 32119 raft_consensus.cc:3037] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.301357 1825 raft_consensus.cc:2781] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:34.301447 1825 raft_consensus.cc:686] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:34.301606 1825 consensus_queue.cc:237] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:34.302179 32119 raft_consensus.cc:2454] T 21a2c1b2947c47289ace8cf41f2fbb4d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:34.303505 29329 catalog_manager.cc:5484] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:34.306248 2231 consensus_queue.cc:1035] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.315867 29327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:34.316032 29327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:34.318172 29327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 02dff3ad4ba84230b5ac52e274310748
I20240502 14:00:34.318693 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 02dff3ad4ba84230b5ac52e274310748 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:34 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:34.319324 32119 raft_consensus.cc:1261] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:00:34.319402 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet 02dff3ad4ba84230b5ac52e274310748 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:34 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:34.319583 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 02dff3ad4ba84230b5ac52e274310748 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:34 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:34.319872 2449 tablet_replica.cc:330] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:34.319926 858 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:34.320047 2449 raft_consensus.cc:2229] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:34.320183 2449 raft_consensus.cc:2258] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.319712 2450 tablet_replica.cc:330] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:34.320775 2449 ts_tablet_manager.cc:1875] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:34.320976 2450 raft_consensus.cc:2229] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:34.321199 2450 raft_consensus.cc:2258] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.322204 2450 ts_tablet_manager.cc:1875] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:34.323119 2449 ts_tablet_manager.cc:1888] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:34.323189 2449 log.cc:1196] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/02dff3ad4ba84230b5ac52e274310748
I20240502 14:00:34.324992 2450 ts_tablet_manager.cc:1888] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:34.325142 2450 log.cc:1196] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/02dff3ad4ba84230b5ac52e274310748
I20240502 14:00:34.325553 2450 ts_tablet_manager.cc:1909] T 02dff3ad4ba84230b5ac52e274310748 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:34.326279 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 02dff3ad4ba84230b5ac52e274310748 (table rebalancer_dynamic_table_new_name [id=eb4d218daa7d44f4955be65395b83945]) successfully deleted
I20240502 14:00:34.327180 2449 ts_tablet_manager.cc:1909] T 02dff3ad4ba84230b5ac52e274310748 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:34.327904 2451 tablet_replica.cc:330] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:34.327908 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 02dff3ad4ba84230b5ac52e274310748 (table rebalancer_dynamic_table_new_name [id=eb4d218daa7d44f4955be65395b83945]) successfully deleted
I20240502 14:00:34.328055 2451 raft_consensus.cc:2229] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:34.328672 2451 raft_consensus.cc:2258] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.329223 2451 ts_tablet_manager.cc:1875] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:34.331403 2451 ts_tablet_manager.cc:1888] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:34.331475 2451 log.cc:1196] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/02dff3ad4ba84230b5ac52e274310748
I20240502 14:00:34.331766 2451 ts_tablet_manager.cc:1909] T 02dff3ad4ba84230b5ac52e274310748 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:34.332460 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 02dff3ad4ba84230b5ac52e274310748 (table rebalancer_dynamic_table_new_name [id=eb4d218daa7d44f4955be65395b83945]) successfully deleted
I20240502 14:00:34.394163 29863 raft_consensus.cc:1261] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:00:34.394662 858 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:34.397501 31990 raft_consensus.cc:1261] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:00:34.398474 2454 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:34.407811 1904 raft_consensus.cc:482] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:34.407950 1904 raft_consensus.cc:504] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.408349 1904 leader_election.cc:288] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:34.408764 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae0eeb814e34729b6284d838d458ddf" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:34.408913 31860 raft_consensus.cc:2454] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:34.408968 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae0eeb814e34729b6284d838d458ddf" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:34.409132 29732 raft_consensus.cc:2454] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:34.409214 32188 leader_election.cc:302] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:34.409376 1904 raft_consensus.cc:2781] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:34.409435 1904 raft_consensus.cc:482] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:34.409478 1904 raft_consensus.cc:3037] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.410359 1904 raft_consensus.cc:504] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.410696 1904 leader_election.cc:288] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:34.411046 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae0eeb814e34729b6284d838d458ddf" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:34.411065 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae0eeb814e34729b6284d838d458ddf" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:34.411168 29732 raft_consensus.cc:3037] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.411167 31860 raft_consensus.cc:3037] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.412055 29732 raft_consensus.cc:2454] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:34.412209 31860 raft_consensus.cc:2454] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:34.412302 32189 leader_election.cc:302] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:34.412461 1904 raft_consensus.cc:2781] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:34.412540 1904 raft_consensus.cc:686] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:34.412676 1904 consensus_queue.cc:237] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:34.414499 29321 catalog_manager.cc:5484] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:34.419801 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:34.424890 31840 tablet_service.cc:1467] Processing CreateTablet for tablet b187b407792b48f5a8a09e604050617d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901]), partition=
I20240502 14:00:34.425055 29713 tablet_service.cc:1467] Processing CreateTablet for tablet b187b407792b48f5a8a09e604050617d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901]), partition=
I20240502 14:00:34.425185 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b187b407792b48f5a8a09e604050617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.425316 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b187b407792b48f5a8a09e604050617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.426833 2395 tablet_bootstrap.cc:492] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:34.426880 2406 tablet_bootstrap.cc:492] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:34.427062 29582 tablet_service.cc:1467] Processing CreateTablet for tablet b187b407792b48f5a8a09e604050617d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901]), partition=
I20240502 14:00:34.427301 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b187b407792b48f5a8a09e604050617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:34.427803 2395 tablet_bootstrap.cc:654] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.427842 2406 tablet_bootstrap.cc:654] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.429345 2395 tablet_bootstrap.cc:492] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:34.429440 2395 ts_tablet_manager.cc:1367] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:34.429490 2399 tablet_bootstrap.cc:492] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:34.429956 2395 raft_consensus.cc:348] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.430063 2395 raft_consensus.cc:374] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.430099 2395 raft_consensus.cc:729] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.430191 2406 tablet_bootstrap.cc:492] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:34.430269 2406 ts_tablet_manager.cc:1367] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:34.430380 2399 tablet_bootstrap.cc:654] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:34.430227 2395 consensus_queue.cc:260] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.430742 2406 raft_consensus.cc:348] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.430889 2406 raft_consensus.cc:374] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.430928 2406 raft_consensus.cc:729] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.430958 2395 ts_tablet_manager.cc:1398] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:00:34.431077 2406 consensus_queue.cc:260] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.431355 2406 ts_tablet_manager.cc:1398] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:34.432134 2399 tablet_bootstrap.cc:492] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:34.432225 2399 ts_tablet_manager.cc:1367] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:34.432678 2399 raft_consensus.cc:348] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.432804 2399 raft_consensus.cc:374] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:34.432844 2399 raft_consensus.cc:729] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:34.432986 2399 consensus_queue.cc:260] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.433285 2399 ts_tablet_manager.cc:1398] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:34.453758 1075 raft_consensus.cc:482] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:34.453888 1075 raft_consensus.cc:504] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.454370 1075 leader_election.cc:288] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:34.454764 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b187b407792b48f5a8a09e604050617d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:34.454766 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b187b407792b48f5a8a09e604050617d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:34.454947 29732 raft_consensus.cc:2454] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:34.454947 29600 raft_consensus.cc:2454] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:34.455242 31794 leader_election.cc:302] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:34.455411 1075 raft_consensus.cc:2781] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:34.455487 1075 raft_consensus.cc:482] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:34.455515 1075 raft_consensus.cc:3037] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.456362 1075 raft_consensus.cc:504] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.456681 1075 leader_election.cc:288] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:34.456992 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b187b407792b48f5a8a09e604050617d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:34.457094 29732 raft_consensus.cc:3037] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.457860 29732 raft_consensus.cc:2454] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:34.458101 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b187b407792b48f5a8a09e604050617d" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:34.458148 31795 leader_election.cc:302] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:34.458230 29600 raft_consensus.cc:3037] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.458302 1075 raft_consensus.cc:2781] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:34.458379 1075 raft_consensus.cc:686] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:34.458518 1075 consensus_queue.cc:237] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:34.459257 29600 raft_consensus.cc:2454] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:34.460397 29327 catalog_manager.cc:5484] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:34.510985 1904 raft_consensus.cc:482] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:34.511123 1904 raft_consensus.cc:504] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.511561 1904 leader_election.cc:288] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:34.512001 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f01d68468f4afca23ca423748ddb58" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:34.512192 32119 raft_consensus.cc:2454] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:34.512260 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f01d68468f4afca23ca423748ddb58" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:34.512419 31860 raft_consensus.cc:2454] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:34.512526 32188 leader_election.cc:302] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:34.512697 1904 raft_consensus.cc:2781] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:34.512769 1904 raft_consensus.cc:482] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:34.512818 1904 raft_consensus.cc:3037] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.513885 1904 raft_consensus.cc:504] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.514250 1904 leader_election.cc:288] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:34.514717 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f01d68468f4afca23ca423748ddb58" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:34.514768 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f01d68468f4afca23ca423748ddb58" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:34.514837 31860 raft_consensus.cc:3037] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.514909 32119 raft_consensus.cc:3037] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:34.515753 31860 raft_consensus.cc:2454] T 54f01d68468f4afca23ca423748ddb58 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:34.515754 32119 raft_consensus.cc:2454] T 54f01d68468f4afca23ca423748ddb58 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:34.516009 32188 leader_election.cc:302] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:34.516165 1904 raft_consensus.cc:2781] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:34.516250 1904 raft_consensus.cc:686] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:34.516386 1904 consensus_queue.cc:237] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:34.518517 29327 catalog_manager.cc:5484] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:34.548497 29582 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:34.548563 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:34.551894 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.555402 31840 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:34.556744 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.565373 32233 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.572149 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:34.578106 29732 raft_consensus.cc:1261] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:34.578305 29600 raft_consensus.cc:1261] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:34.578727 2454 consensus_queue.cc:1035] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.578872 1075 consensus_queue.cc:1035] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.583987 29327 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet b187b407792b48f5a8a09e604050617d (table rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901]). Expected version 1 got 0
I20240502 14:00:34.584540 32103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:34.585403 29327 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet b187b407792b48f5a8a09e604050617d (table rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901]). Expected version 1 got 0
I20240502 14:00:34.587591 2305 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:34.587927 2305 tablet.cc:1620] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:34.589006 29327 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet b187b407792b48f5a8a09e604050617d (table rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901]). Expected version 1 got 0
I20240502 14:00:34.594280 2479 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:34.594732 2479 tablet.cc:1620] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:34.600301 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901] alter complete (version 1)
I20240502 14:00:34.612494 2480 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:34.612887 2480 tablet.cc:1620] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:34.613452 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:34.615024 29327 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet b187b407792b48f5a8a09e604050617d (table rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901]). Expected version 1 got 0
I20240502 14:00:34.652839 32119 consensus_queue.cc:237] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:34.654079 29471 raft_consensus.cc:1261] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:34.654106 32254 raft_consensus.cc:1261] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:34.654490 2231 consensus_queue.cc:1035] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:34.654696 2447 consensus_queue.cc:1035] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:34.656683 2447 raft_consensus.cc:2932] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:34.657088 29471 raft_consensus.cc:2932] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:34.657480 32254 raft_consensus.cc:2932] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:34.660224 29321 catalog_manager.cc:5484] T 70367c7c718d4d9195f0af7e69bdf2ec P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:34.663833 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 70367c7c718d4d9195f0af7e69bdf2ec with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:34.664271 2505 tablet_replica.cc:330] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:34.664530 2505 raft_consensus.cc:2229] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:34.664798 2505 raft_consensus.cc:2258] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.665482 2505 ts_tablet_manager.cc:1875] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:34.667913 2505 ts_tablet_manager.cc:1888] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:34.667992 2505 log.cc:1196] T 70367c7c718d4d9195f0af7e69bdf2ec P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/70367c7c718d4d9195f0af7e69bdf2ec
I20240502 14:00:34.668478 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 70367c7c718d4d9195f0af7e69bdf2ec (table rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]) successfully deleted
I20240502 14:00:34.707020 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:34.715517 2479 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:34.715860 2479 tablet.cc:1620] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:34.716867 2480 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:34.717144 2480 tablet.cc:1620] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:34.719404 2305 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:34.719696 2305 tablet.cc:1620] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:34.722771 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=af74abd5099f4605b25019bde05f4901] alter complete (version 2)
I20240502 14:00:34.748154 1825 consensus_queue.cc:1035] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.754688 1537 consensus_queue.cc:1035] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.763664 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.770447 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:34.770769 29582 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:34.775830 32233 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.783006 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:34.796613 31840 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:34.797950 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.812223 32103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:34.825215 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:34.831480 2479 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:34.835211 2305 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:34.836257 2480 tablet.cc:1722] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:34.838907 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=af74abd5099f4605b25019bde05f4901] alter complete (version 3)
I20240502 14:00:34.861004 32119 consensus_queue.cc:237] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:34.862316 29862 raft_consensus.cc:1261] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:34.862305 32254 raft_consensus.cc:1261] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:34.862700 2447 consensus_queue.cc:1035] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:34.862897 2231 consensus_queue.cc:1035] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:34.864601 2231 raft_consensus.cc:2932] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:34.865023 29862 raft_consensus.cc:2932] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:34.865773 32254 raft_consensus.cc:2932] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:34.867393 29322 catalog_manager.cc:5484] T 82cda992fe784d9f9015e6a3d431f1cd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:34.870618 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cda992fe784d9f9015e6a3d431f1cd with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:34.870981 2586 tablet_replica.cc:330] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:34.871145 2586 raft_consensus.cc:2229] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:34.871318 2586 raft_consensus.cc:2258] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.871783 2586 ts_tablet_manager.cc:1875] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:34.874408 2586 ts_tablet_manager.cc:1888] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:34.874480 2586 log.cc:1196] T 82cda992fe784d9f9015e6a3d431f1cd P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/82cda992fe784d9f9015e6a3d431f1cd
I20240502 14:00:34.875030 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 82cda992fe784d9f9015e6a3d431f1cd (table rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]) successfully deleted
I20240502 14:00:34.935982 1904 consensus_queue.cc:1035] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.940563 1941 consensus_queue.cc:1035] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:34.941047 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:34.941176 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:34.944952 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet b187b407792b48f5a8a09e604050617d
I20240502 14:00:34.945520 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet b187b407792b48f5a8a09e604050617d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:34 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:34.945581 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet b187b407792b48f5a8a09e604050617d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:34 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:34.945722 2505 tablet_replica.cc:330] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:34.945859 2505 raft_consensus.cc:2229] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:34.946048 2505 raft_consensus.cc:2258] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.946363 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet b187b407792b48f5a8a09e604050617d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:34 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:34.946985 2505 ts_tablet_manager.cc:1875] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:34.949261 2505 ts_tablet_manager.cc:1888] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:34.949334 2505 log.cc:1196] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/b187b407792b48f5a8a09e604050617d
I20240502 14:00:34.949636 2505 ts_tablet_manager.cc:1909] T b187b407792b48f5a8a09e604050617d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:34.956878 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet b187b407792b48f5a8a09e604050617d (table rebalancer_dynamic_table_new_name [id=af74abd5099f4605b25019bde05f4901]) successfully deleted
I20240502 14:00:34.961782 2642 tablet_replica.cc:330] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:34.962018 2642 raft_consensus.cc:2229] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:34.962200 2642 raft_consensus.cc:2258] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.962960 2642 ts_tablet_manager.cc:1875] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:34.964797 2641 tablet_replica.cc:330] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:34.965003 2641 raft_consensus.cc:2229] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:34.965193 2641 raft_consensus.cc:2258] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:34.965378 2642 ts_tablet_manager.cc:1888] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:34.965512 2642 log.cc:1196] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/b187b407792b48f5a8a09e604050617d
I20240502 14:00:34.965875 2642 ts_tablet_manager.cc:1909] T b187b407792b48f5a8a09e604050617d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:34.966341 2641 ts_tablet_manager.cc:1875] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:34.966554 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet b187b407792b48f5a8a09e604050617d (table rebalancer_dynamic_table_new_name [id=af74abd5099f4605b25019bde05f4901]) successfully deleted
I20240502 14:00:34.968919 2641 ts_tablet_manager.cc:1888] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:34.969005 2641 log.cc:1196] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/b187b407792b48f5a8a09e604050617d
I20240502 14:00:34.973634 2641 ts_tablet_manager.cc:1909] T b187b407792b48f5a8a09e604050617d P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:34.974485 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet b187b407792b48f5a8a09e604050617d (table rebalancer_dynamic_table_new_name [id=af74abd5099f4605b25019bde05f4901]) successfully deleted
I20240502 14:00:34.986574 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:34.988009 29582 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:34.991086 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.000552 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.001998 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:35.011730 31840 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:35.015980 32233 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.033201 32103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:35.034072 29324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:35.042604 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 995449d84e4b4d4692f8f225b5a44830 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.043016 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995449d84e4b4d4692f8f225b5a44830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.043248 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 995449d84e4b4d4692f8f225b5a44830 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.043532 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995449d84e4b4d4692f8f225b5a44830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.046583 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 4c08999e4e7f4d0bb6a03199ee676f76 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.046900 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c08999e4e7f4d0bb6a03199ee676f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.046980 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 995449d84e4b4d4692f8f225b5a44830 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.047153 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 4c08999e4e7f4d0bb6a03199ee676f76 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.047260 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995449d84e4b4d4692f8f225b5a44830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.047446 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c08999e4e7f4d0bb6a03199ee676f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.049536 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 6df051496f954569983892bfe828f653 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.049784 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df051496f954569983892bfe828f653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.049876 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 6df051496f954569983892bfe828f653 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.050134 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df051496f954569983892bfe828f653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.051476 2658 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:35.051925 2654 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:35.052632 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 6df051496f954569983892bfe828f653 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.052917 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df051496f954569983892bfe828f653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.053197 29324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:35.055177 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 4c08999e4e7f4d0bb6a03199ee676f76 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.055433 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c08999e4e7f4d0bb6a03199ee676f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.055825 2654 tablet_bootstrap.cc:654] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.057307 2656 tablet_bootstrap.cc:492] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:35.057561 2661 tablet_bootstrap.cc:492] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:35.058144 2656 tablet_bootstrap.cc:654] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.058445 2661 tablet_bootstrap.cc:654] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.059089 2657 tablet_bootstrap.cc:492] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:35.059509 2654 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:35.059597 2654 ts_tablet_manager.cc:1367] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:35.060237 2659 tablet_bootstrap.cc:492] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:35.060372 2655 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:35.061424 2657 tablet_bootstrap.cc:654] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.060066 2654 raft_consensus.cc:348] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.061740 2654 raft_consensus.cc:374] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.061831 2654 raft_consensus.cc:729] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.062527 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 755c7815f5b74b33a8d252a96adb4fa8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54]), partition=
I20240502 14:00:35.062564 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 755c7815f5b74b33a8d252a96adb4fa8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54]), partition=
I20240502 14:00:35.062772 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755c7815f5b74b33a8d252a96adb4fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.062798 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755c7815f5b74b33a8d252a96adb4fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.063020 2658 tablet_bootstrap.cc:654] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.064270 2661 tablet_bootstrap.cc:492] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:35.064368 2661 ts_tablet_manager.cc:1367] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:35.064884 2661 raft_consensus.cc:348] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.065061 2661 raft_consensus.cc:374] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.065135 2661 raft_consensus.cc:729] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.062057 2654 consensus_queue.cc:260] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.065325 2661 consensus_queue.cc:260] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.065513 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 755c7815f5b74b33a8d252a96adb4fa8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54]), partition=
I20240502 14:00:35.065752 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755c7815f5b74b33a8d252a96adb4fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.066838 2656 tablet_bootstrap.cc:492] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:35.066947 2656 ts_tablet_manager.cc:1367] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:35.067448 2661 ts_tablet_manager.cc:1398] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:35.068543 2655 tablet_bootstrap.cc:654] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.068795 2659 tablet_bootstrap.cc:654] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.068987 2656 raft_consensus.cc:348] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.065510 2654 ts_tablet_manager.cc:1398] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:35.069304 2656 raft_consensus.cc:374] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.069365 2654 tablet_bootstrap.cc:492] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:35.069866 2656 raft_consensus.cc:729] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.070392 2656 consensus_queue.cc:260] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.070749 2656 ts_tablet_manager.cc:1398] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:35.070950 2656 tablet_bootstrap.cc:492] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:35.071987 2656 tablet_bootstrap.cc:654] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.071993 2654 tablet_bootstrap.cc:654] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.073238 2657 tablet_bootstrap.cc:492] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:35.073333 2656 tablet_bootstrap.cc:492] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:35.073383 2657 ts_tablet_manager.cc:1367] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:35.073432 2656 ts_tablet_manager.cc:1367] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:35.073884 2657 raft_consensus.cc:348] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.074560 2665 tablet_bootstrap.cc:492] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:35.074577 2656 raft_consensus.cc:348] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.075052 2656 raft_consensus.cc:374] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.075150 2656 raft_consensus.cc:729] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.074038 2657 raft_consensus.cc:374] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.075362 2657 raft_consensus.cc:729] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.075373 2656 consensus_queue.cc:260] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.075584 2657 consensus_queue.cc:260] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.076025 2657 ts_tablet_manager.cc:1398] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:35.076093 2655 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:35.076224 2655 ts_tablet_manager.cc:1367] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:00:35.076787 2659 tablet_bootstrap.cc:492] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:35.076921 2659 ts_tablet_manager.cc:1367] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:35.077364 2656 ts_tablet_manager.cc:1398] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:35.077593 2656 tablet_bootstrap.cc:492] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:35.077646 2659 raft_consensus.cc:348] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.077819 2659 raft_consensus.cc:374] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.077905 2659 raft_consensus.cc:729] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.078091 2659 consensus_queue.cc:260] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.078531 2656 tablet_bootstrap.cc:654] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.078886 2655 raft_consensus.cc:348] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.079087 2655 raft_consensus.cc:374] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.079208 2655 raft_consensus.cc:729] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.079737 2655 consensus_queue.cc:260] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.080022 2659 ts_tablet_manager.cc:1398] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:35.080264 2655 ts_tablet_manager.cc:1398] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:35.080423 2655 tablet_bootstrap.cc:492] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:35.081880 2655 tablet_bootstrap.cc:654] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.082947 2665 tablet_bootstrap.cc:654] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.085338 2665 tablet_bootstrap.cc:492] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:35.085484 2665 ts_tablet_manager.cc:1367] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240502 14:00:35.086086 2655 tablet_bootstrap.cc:492] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:35.086206 2655 ts_tablet_manager.cc:1367] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:35.086699 2655 raft_consensus.cc:348] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.087463 2654 tablet_bootstrap.cc:492] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:35.087563 2654 ts_tablet_manager.cc:1367] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:00:35.087584 2665 raft_consensus.cc:348] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.087699 2665 raft_consensus.cc:374] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.087797 2655 raft_consensus.cc:374] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.087884 2655 raft_consensus.cc:729] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.088588 2654 raft_consensus.cc:348] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.088698 2654 raft_consensus.cc:374] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.088730 2654 raft_consensus.cc:729] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.088737 2656 tablet_bootstrap.cc:492] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:35.088824 2656 ts_tablet_manager.cc:1367] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:35.088896 2654 consensus_queue.cc:260] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.088243 2655 consensus_queue.cc:260] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.089226 1941 consensus_queue.cc:1035] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.089293 2656 raft_consensus.cc:348] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.089406 2656 raft_consensus.cc:374] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.089444 2656 raft_consensus.cc:729] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.089563 2656 consensus_queue.cc:260] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.089594 2654 ts_tablet_manager.cc:1398] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:35.089812 2656 ts_tablet_manager.cc:1398] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.089358 2665 raft_consensus.cc:729] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.090831 2655 ts_tablet_manager.cc:1398] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:35.091035 2665 consensus_queue.cc:260] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.092885 2665 ts_tablet_manager.cc:1398] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20240502 14:00:35.094403 2658 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:35.094508 2658 ts_tablet_manager.cc:1367] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.002s
I20240502 14:00:35.095057 2658 raft_consensus.cc:348] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.095188 2658 raft_consensus.cc:374] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.095227 2658 raft_consensus.cc:729] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.095340 2658 consensus_queue.cc:260] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.095602 2658 ts_tablet_manager.cc:1398] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.100648 2680 consensus_queue.cc:1035] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.114149 29862 consensus_queue.cc:237] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:35.115574 32119 raft_consensus.cc:1261] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:35.115659 31990 raft_consensus.cc:1261] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:35.116214 2165 consensus_queue.cc:1035] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:35.116604 2165 consensus_queue.cc:1035] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:35.118754 2165 raft_consensus.cc:2932] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.119446 31992 raft_consensus.cc:2932] T 9790e7ee6d8945bf85f129ef3f9900ca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.121810 29327 catalog_manager.cc:5484] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.123193 32119 raft_consensus.cc:2932] T 9790e7ee6d8945bf85f129ef3f9900ca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.126994 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 9790e7ee6d8945bf85f129ef3f9900ca with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:35.127219 2586 tablet_replica.cc:330] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:35.127368 2586 raft_consensus.cc:2229] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:35.127493 2586 raft_consensus.cc:2258] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:35.127964 2586 ts_tablet_manager.cc:1875] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:35.129853 2586 ts_tablet_manager.cc:1888] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:35.129941 2586 log.cc:1196] T 9790e7ee6d8945bf85f129ef3f9900ca P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/9790e7ee6d8945bf85f129ef3f9900ca
I20240502 14:00:35.130493 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 9790e7ee6d8945bf85f129ef3f9900ca (table rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]) successfully deleted
I20240502 14:00:35.180905 29862 consensus_queue.cc:237] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:35.182207 29600 raft_consensus.cc:1261] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:35.182416 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59f4faff46704ca1837edd139596d9e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.182412 29732 raft_consensus.cc:1261] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.182621 2737 consensus_queue.cc:1035] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.182780 2165 consensus_queue.cc:1035] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.184902 2165 raft_consensus.cc:2932] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.185353 29600 raft_consensus.cc:2932] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.187268 29732 raft_consensus.cc:2932] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.189481 29327 catalog_manager.cc:5484] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:35.194820 2231 raft_consensus.cc:482] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:35.194958 2231 raft_consensus.cc:504] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.195385 2231 leader_election.cc:288] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:35.195726 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df051496f954569983892bfe828f653" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:35.195875 32254 raft_consensus.cc:2454] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:35.196166 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df051496f954569983892bfe828f653" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:35.196312 31992 raft_consensus.cc:2454] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:35.196616 32056 leader_election.cc:302] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:35.198168 2231 raft_consensus.cc:2781] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:35.198237 2231 raft_consensus.cc:482] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:35.198280 2231 raft_consensus.cc:3037] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.199303 2231 raft_consensus.cc:504] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.199667 2231 leader_election.cc:288] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:35.199952 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df051496f954569983892bfe828f653" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:35.200011 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df051496f954569983892bfe828f653" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:35.200063 32254 raft_consensus.cc:3037] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.200137 31992 raft_consensus.cc:3037] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.200951 32254 raft_consensus.cc:2454] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:35.201246 32058 leader_election.cc:302] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:35.201408 2231 raft_consensus.cc:2781] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:35.201483 2231 raft_consensus.cc:686] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:35.201614 2231 consensus_queue.cc:237] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.201649 31992 raft_consensus.cc:2454] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:35.203687 29320 catalog_manager.cc:5484] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:35.209806 29600 consensus_queue.cc:237] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.211195 29732 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:35.211280 29535 consensus_peers.cc:485] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d2ab1ded2c241038d5685678aa819a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.211584 29862 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.211570 1537 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.211964 1537 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.213614 1825 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:35.213999 29862 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:35.215811 29732 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:35.216754 29329 catalog_manager.cc:5484] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:35.235565 29732 consensus_queue.cc:237] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.237147 29471 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:35.237214 29666 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fc14b4bd42841df91972e0c5d02faa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.237530 29600 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.237525 2382 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.237902 2778 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.240180 2382 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.240578 29600 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.242336 29470 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.242911 29320 catalog_manager.cc:5484] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.254375 1537 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:35.254484 1537 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.254921 1537 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:35.255210 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995449d84e4b4d4692f8f225b5a44830" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:35.255290 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995449d84e4b4d4692f8f225b5a44830" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:35.255393 31992 raft_consensus.cc:2454] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:35.255401 29862 raft_consensus.cc:2454] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:35.255656 29535 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:35.255808 1537 raft_consensus.cc:2781] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:35.255864 1537 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:35.255918 1537 raft_consensus.cc:3037] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.257057 1537 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.257745 1537 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:35.258054 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995449d84e4b4d4692f8f225b5a44830" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:35.258117 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995449d84e4b4d4692f8f225b5a44830" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:35.258191 29862 raft_consensus.cc:3037] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.258265 31992 raft_consensus.cc:3037] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.259140 29862 raft_consensus.cc:2454] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:35.259351 31992 raft_consensus.cc:2454] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:35.259563 29536 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:35.259735 1537 raft_consensus.cc:2781] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:35.259807 1537 raft_consensus.cc:686] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:35.259939 1537 consensus_queue.cc:237] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.262207 29329 catalog_manager.cc:5484] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:35.266947 32119 consensus_queue.cc:237] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.267087 858 raft_consensus.cc:482] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:35.267187 858 raft_consensus.cc:504] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.267590 858 leader_election.cc:288] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:35.267902 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755c7815f5b74b33a8d252a96adb4fa8" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:35.268003 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755c7815f5b74b33a8d252a96adb4fa8" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:35.268040 29862 raft_consensus.cc:2454] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:35.268121 32119 raft_consensus.cc:2454] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:35.268524 31795 leader_election.cc:302] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:35.268626 31992 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:35.268743 32057 consensus_peers.cc:485] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1116d9f165b488796ab466f9a866471. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.268863 858 raft_consensus.cc:2781] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:35.268916 858 raft_consensus.cc:482] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:35.268954 858 raft_consensus.cc:3037] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.269006 32254 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:35.269445 2447 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:35.269941 858 raft_consensus.cc:504] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.270275 858 leader_election.cc:288] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:35.270655 2447 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:35.270905 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755c7815f5b74b33a8d252a96adb4fa8" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:35.270967 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755c7815f5b74b33a8d252a96adb4fa8" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:35.271009 32119 raft_consensus.cc:3037] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.271070 29862 raft_consensus.cc:3037] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.271957 32119 raft_consensus.cc:2454] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:35.271961 29862 raft_consensus.cc:2454] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:35.272243 31795 leader_election.cc:302] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:35.272420 858 raft_consensus.cc:2781] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:35.272507 858 raft_consensus.cc:686] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:35.272641 858 consensus_queue.cc:237] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:35.272866 2447 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:35.275260 32254 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:35.274755 31990 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:35.275589 29327 catalog_manager.cc:5484] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:35.275605 29329 catalog_manager.cc:5484] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:35.302459 32119 consensus_queue.cc:237] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:35.304442 31860 raft_consensus.cc:1261] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:35.304504 32056 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d3a84c1d004412db381854147330dfc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.304950 31990 raft_consensus.cc:1261] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:35.305146 2447 consensus_queue.cc:1035] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:35.305320 2798 consensus_queue.cc:1035] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:35.311599 2798 raft_consensus.cc:2932] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:35.311970 31860 raft_consensus.cc:2932] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:35.313943 31990 raft_consensus.cc:2932] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:35.314553 29320 catalog_manager.cc:5484] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:35.328056 2447 raft_consensus.cc:482] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:35.328168 2447 raft_consensus.cc:504] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.328510 2447 leader_election.cc:288] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:35.328838 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c08999e4e7f4d0bb6a03199ee676f76" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:35.329006 29470 raft_consensus.cc:2454] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:35.329288 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c08999e4e7f4d0bb6a03199ee676f76" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:35.329342 32058 leader_election.cc:302] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:35.329423 29732 raft_consensus.cc:2454] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:35.329504 2821 raft_consensus.cc:2781] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:35.329545 2821 raft_consensus.cc:482] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:35.329571 2821 raft_consensus.cc:3037] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.330413 2821 raft_consensus.cc:504] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.330760 2821 leader_election.cc:288] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:35.331224 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c08999e4e7f4d0bb6a03199ee676f76" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:35.331353 29732 raft_consensus.cc:3037] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.332298 29732 raft_consensus.cc:2454] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:35.332571 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c08999e4e7f4d0bb6a03199ee676f76" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:35.332609 32057 leader_election.cc:302] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:35.332700 29470 raft_consensus.cc:3037] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:35.332995 2821 raft_consensus.cc:2781] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:35.333094 2821 raft_consensus.cc:686] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:35.333235 2821 consensus_queue.cc:237] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.333590 29470 raft_consensus.cc:2454] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:35.335309 29327 catalog_manager.cc:5484] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:35.343634 31990 consensus_queue.cc:237] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.345187 29470 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:35.345417 32119 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:35.345935 2300 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:35.346138 2839 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:00:35.346684 31927 consensus_peers.cc:485] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec8e4ac081604ec8a3f463f740d9119c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.349328 2300 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:35.349797 29470 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:35.351171 32119 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:35.353410 29327 catalog_manager.cc:5484] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:35.375455 29601 consensus_queue.cc:237] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.376742 29470 raft_consensus.cc:1261] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.376742 29732 raft_consensus.cc:1261] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.377210 1825 consensus_queue.cc:1035] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.377391 2758 consensus_queue.cc:1035] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:35.377559 29535 consensus_peers.cc:485] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6eae206e12b44b809d8b2fcc21611a28. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.379192 2758 raft_consensus.cc:2932] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.379611 29732 raft_consensus.cc:2932] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.381408 29470 raft_consensus.cc:2932] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.383569 29329 catalog_manager.cc:5484] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:35.408694 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:35.412592 32119 raft_consensus.cc:1261] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:35.412592 29862 raft_consensus.cc:1261] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:35.413079 858 consensus_queue.cc:1035] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.413268 1075 consensus_queue.cc:1035] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.415906 2882 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:35.415992 29320 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet 755c7815f5b74b33a8d252a96adb4fa8 (table rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54]). Expected version 1 got 0
I20240502 14:00:35.416623 2882 tablet.cc:1620] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:35.416682 2883 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:35.416980 2883 tablet.cc:1620] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:35.420249 29327 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet 755c7815f5b74b33a8d252a96adb4fa8 (table rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54]). Expected version 1 got 0
I20240502 14:00:35.420536 2884 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:35.420640 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 755c7815f5b74b33a8d252a96adb4fa8 (table rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54]). Expected version 1 got 0
I20240502 14:00:35.420923 2884 tablet.cc:1620] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:35.425124 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54] alter complete (version 1)
I20240502 14:00:35.473201 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.479060 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:35.481696 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:35.483750 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.491694 32233 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.493542 29582 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:35.514484 31840 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:35.515050 32101 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:35.528317 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:35.535189 2882 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:35.535521 2882 tablet.cc:1620] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:35.539342 2883 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:35.539674 2883 tablet.cc:1620] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:35.544266 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0e25e21c7f6347cbb192fe0a679fda54] alter complete (version 2)
I20240502 14:00:35.545562 2884 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:35.545857 2884 tablet.cc:1620] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:35.613111 2962 ts_tablet_manager.cc:902] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:35.614395 2962 tablet_copy_client.cc:308] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:35.621872 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 8d2ab1ded2c241038d5685678aa819a8 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:52525)
I20240502 14:00:35.622030 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 8d2ab1ded2c241038d5685678aa819a8 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:52525: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-8d2ab1ded2c241038d5685678aa819a8
I20240502 14:00:35.623143 29622 tablet_copy_source_session.cc:205] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:35.624415 2962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2ab1ded2c241038d5685678aa819a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.627143 2962 tablet_copy_client.cc:783] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:35.627404 2962 tablet_copy_client.cc:647] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:35.628948 2962 tablet_copy_client.cc:523] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:35.630456 2962 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:35.647094 2980 ts_tablet_manager.cc:902] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:35.647848 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.647854 2980 tablet_copy_client.cc:308] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:35.649488 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:35.649866 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:35.650655 29582 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:35.653745 31840 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:35.654307 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 59f4faff46704ca1837edd139596d9e2 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37167)
I20240502 14:00:35.654424 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 59f4faff46704ca1837edd139596d9e2 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37167: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-59f4faff46704ca1837edd139596d9e2
I20240502 14:00:35.655436 29884 tablet_copy_source_session.cc:205] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:35.656713 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.664145 2980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f4faff46704ca1837edd139596d9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.668629 2821 consensus_queue.cc:1035] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.669574 2882 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:35.669876 2962 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:35.670163 2962 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:35.670310 2962 ts_tablet_manager.cc:1367] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.000s
I20240502 14:00:35.670670 2962 raft_consensus.cc:348] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.670756 2962 raft_consensus.cc:729] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.670909 2962 consensus_queue.cc:260] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:35.671219 2962 ts_tablet_manager.cc:1398] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:00:35.672667 2884 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:35.676363 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-8d2ab1ded2c241038d5685678aa819a8 received from {username='slave'} at 127.22.55.134:52525
I20240502 14:00:35.676470 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-8d2ab1ded2c241038d5685678aa819a8 on tablet 8d2ab1ded2c241038d5685678aa819a8 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:35.680048 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=0e25e21c7f6347cbb192fe0a679fda54] alter complete (version 3)
I20240502 14:00:35.683465 32233 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.683955 2883 tablet.cc:1722] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:35.687202 2231 consensus_queue.cc:1035] T 6df051496f954569983892bfe828f653 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.691897 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:35.695034 32101 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:35.695946 2980 tablet_copy_client.cc:783] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:35.706806 2980 tablet_copy_client.cc:647] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:35.708077 2980 tablet_copy_client.cc:523] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:35.710117 2980 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:35.715646 2231 consensus_queue.cc:1035] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.744431 2231 consensus_queue.cc:1035] T 4c08999e4e7f4d0bb6a03199ee676f76 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.746338 2993 ts_tablet_manager.cc:902] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:35.746920 2993 tablet_copy_client.cc:308] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:35.747455 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet a1116d9f165b488796ab466f9a866471 from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:33761)
I20240502 14:00:35.747548 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet a1116d9f165b488796ab466f9a866471 from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:33761: session id = efdbf7bdfe0b41bdadb1add053b09a7e-a1116d9f165b488796ab466f9a866471
I20240502 14:00:35.748423 32143 tablet_copy_source_session.cc:205] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:35.749336 2993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116d9f165b488796ab466f9a866471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.751850 2993 tablet_copy_client.cc:783] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:35.752039 2993 tablet_copy_client.cc:647] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:35.753105 2993 tablet_copy_client.cc:523] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:35.754832 2993 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:35.755227 2980 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:35.755542 2980 tablet_bootstrap.cc:492] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:35.755710 2980 ts_tablet_manager.cc:1367] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.046s user 0.027s sys 0.001s
I20240502 14:00:35.756156 2980 raft_consensus.cc:348] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:35.756275 2980 raft_consensus.cc:729] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.756423 2980 consensus_queue.cc:260] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:35.756764 2980 ts_tablet_manager.cc:1398] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.757207 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-59f4faff46704ca1837edd139596d9e2 received from {username='slave'} at 127.22.55.135:37167
I20240502 14:00:35.757304 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-59f4faff46704ca1837edd139596d9e2 on tablet 59f4faff46704ca1837edd139596d9e2 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:35.760481 2993 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:35.760718 2993 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:00:35.760872 2993 ts_tablet_manager.cc:1367] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 14:00:35.761322 2993 raft_consensus.cc:348] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.761432 2993 raft_consensus.cc:729] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.761595 2993 consensus_queue.cc:260] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:35.761893 2993 ts_tablet_manager.cc:1398] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:35.762538 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-a1116d9f165b488796ab466f9a866471 received from {username='slave'} at 127.22.55.131:33761
I20240502 14:00:35.762617 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-a1116d9f165b488796ab466f9a866471 on tablet a1116d9f165b488796ab466f9a866471 with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:35.781764 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:35.781924 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:35.784014 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 755c7815f5b74b33a8d252a96adb4fa8
I20240502 14:00:35.784658 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 755c7815f5b74b33a8d252a96adb4fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:35 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:35.784791 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 755c7815f5b74b33a8d252a96adb4fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:35 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:35.784972 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 755c7815f5b74b33a8d252a96adb4fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:35 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:35.785149 3012 tablet_replica.cc:330] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:35.785292 3012 raft_consensus.cc:2229] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:35.785343 3015 tablet_replica.cc:330] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:35.785393 3014 tablet_replica.cc:330] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:35.785517 3015 raft_consensus.cc:2229] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:35.785591 3012 raft_consensus.cc:2258] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:35.785596 3014 raft_consensus.cc:2229] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:35.785704 3014 raft_consensus.cc:2258] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:35.785758 3015 raft_consensus.cc:2258] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:35.786319 3014 ts_tablet_manager.cc:1875] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:35.786449 3012 ts_tablet_manager.cc:1875] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:35.786554 3015 ts_tablet_manager.cc:1875] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:35.788841 3014 ts_tablet_manager.cc:1888] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:35.788962 3014 log.cc:1196] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/755c7815f5b74b33a8d252a96adb4fa8
I20240502 14:00:35.788964 3015 ts_tablet_manager.cc:1888] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:35.788962 3012 ts_tablet_manager.cc:1888] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:35.789028 3012 log.cc:1196] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/755c7815f5b74b33a8d252a96adb4fa8
I20240502 14:00:35.789028 3015 log.cc:1196] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/755c7815f5b74b33a8d252a96adb4fa8
I20240502 14:00:35.789326 3015 ts_tablet_manager.cc:1909] T 755c7815f5b74b33a8d252a96adb4fa8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:35.789326 3012 ts_tablet_manager.cc:1909] T 755c7815f5b74b33a8d252a96adb4fa8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:35.789434 3014 ts_tablet_manager.cc:1909] T 755c7815f5b74b33a8d252a96adb4fa8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:35.790040 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 755c7815f5b74b33a8d252a96adb4fa8 (table rebalancer_dynamic_table_new_name [id=0e25e21c7f6347cbb192fe0a679fda54]) successfully deleted
I20240502 14:00:35.790114 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 755c7815f5b74b33a8d252a96adb4fa8 (table rebalancer_dynamic_table_new_name [id=0e25e21c7f6347cbb192fe0a679fda54]) successfully deleted
I20240502 14:00:35.790185 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 755c7815f5b74b33a8d252a96adb4fa8 (table rebalancer_dynamic_table_new_name [id=0e25e21c7f6347cbb192fe0a679fda54]) successfully deleted
I20240502 14:00:35.797081 1825 consensus_queue.cc:1035] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.800457 1537 consensus_queue.cc:1035] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:35.833600 3049 ts_tablet_manager.cc:902] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:35.834156 3049 tablet_copy_client.cc:308] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:35.834646 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 1fc14b4bd42841df91972e0c5d02faa4 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:57583)
I20240502 14:00:35.834758 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 1fc14b4bd42841df91972e0c5d02faa4 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:57583: session id = f0e01c4f581843e9ad4ced97f0072dc4-1fc14b4bd42841df91972e0c5d02faa4
I20240502 14:00:35.835793 29753 tablet_copy_source_session.cc:205] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:35.837209 3049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc14b4bd42841df91972e0c5d02faa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.839417 3049 tablet_copy_client.cc:783] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:35.839569 3049 tablet_copy_client.cc:647] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:35.841722 3049 tablet_copy_client.cc:523] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:35.843030 3049 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:35.853916 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:35.857205 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.859073 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:35.860328 31840 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:35.860801 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:35.867041 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.874475 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7bfc5231a542dda0a51aee4543299d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.874778 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7bfc5231a542dda0a51aee4543299d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.875336 32101 tablet_service.cc:1467] Processing CreateTablet for tablet fe66bde51c1d4edfa739f6b9a833b129 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.875531 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 891851115e2e4d9db5bf3a0013b0edba (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.875543 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 891851115e2e4d9db5bf3a0013b0edba (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.875623 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe66bde51c1d4edfa739f6b9a833b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.875738 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 891851115e2e4d9db5bf3a0013b0edba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.875809 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 891851115e2e4d9db5bf3a0013b0edba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.877939 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7bfc5231a542dda0a51aee4543299d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.878194 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7bfc5231a542dda0a51aee4543299d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.878302 32233 tablet_service.cc:1467] Processing CreateTablet for tablet fe66bde51c1d4edfa739f6b9a833b129 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.878345 29580 tablet_service.cc:1467] Processing CreateTablet for tablet fe66bde51c1d4edfa739f6b9a833b129 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.878574 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe66bde51c1d4edfa739f6b9a833b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.878618 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe66bde51c1d4edfa739f6b9a833b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.878353 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7bfc5231a542dda0a51aee4543299d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.880270 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 891851115e2e4d9db5bf3a0013b0edba (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=adf24b35b3104baf83041993a19e6425]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:35.880513 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 891851115e2e4d9db5bf3a0013b0edba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.881866 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7bfc5231a542dda0a51aee4543299d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.893285 3067 tablet_bootstrap.cc:492] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:35.896314 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:35.896709 3067 tablet_bootstrap.cc:654] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.898193 3068 tablet_bootstrap.cc:492] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:35.899614 3070 tablet_bootstrap.cc:492] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:35.899925 3069 tablet_bootstrap.cc:492] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:35.900198 3071 tablet_bootstrap.cc:492] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:35.901265 3070 tablet_bootstrap.cc:654] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.902304 3069 tablet_bootstrap.cc:654] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.902796 3068 tablet_bootstrap.cc:654] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.903822 3071 tablet_bootstrap.cc:654] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.905187 3066 tablet_bootstrap.cc:492] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:35.906836 3066 tablet_bootstrap.cc:654] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.909440 3069 tablet_bootstrap.cc:492] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:35.909749 3069 ts_tablet_manager.cc:1367] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:35.910301 3069 raft_consensus.cc:348] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.910507 3069 raft_consensus.cc:374] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.910579 3069 raft_consensus.cc:729] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.910789 3069 consensus_queue.cc:260] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.911237 3069 ts_tablet_manager.cc:1398] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.912361 3068 tablet_bootstrap.cc:492] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:35.912535 3070 tablet_bootstrap.cc:492] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:35.912571 3068 ts_tablet_manager.cc:1367] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:35.913139 3070 ts_tablet_manager.cc:1367] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:35.913702 3068 raft_consensus.cc:348] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.913882 3068 raft_consensus.cc:374] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.913954 3068 raft_consensus.cc:729] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.914129 3068 consensus_queue.cc:260] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.914520 3068 ts_tablet_manager.cc:1398] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:35.915081 3068 tablet_bootstrap.cc:492] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:35.916085 3068 tablet_bootstrap.cc:654] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.918385 3070 raft_consensus.cc:348] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.918576 3070 raft_consensus.cc:374] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.918644 3070 raft_consensus.cc:729] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.918818 3070 consensus_queue.cc:260] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.919703 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 5263a15b9461402ebba6425a890e22c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998]), partition=
I20240502 14:00:35.919697 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 5263a15b9461402ebba6425a890e22c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998]), partition=
I20240502 14:00:35.919972 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5263a15b9461402ebba6425a890e22c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.919968 3070 ts_tablet_manager.cc:1398] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:35.920181 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5263a15b9461402ebba6425a890e22c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.920738 3071 tablet_bootstrap.cc:492] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:35.920822 3071 ts_tablet_manager.cc:1367] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240502 14:00:35.921346 3071 raft_consensus.cc:348] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:35.921494 3071 raft_consensus.cc:374] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.921527 3071 raft_consensus.cc:729] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.921682 3071 consensus_queue.cc:260] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:35.922088 3071 ts_tablet_manager.cc:1398] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.922268 3071 tablet_bootstrap.cc:492] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:35.922565 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 5263a15b9461402ebba6425a890e22c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998]), partition=
I20240502 14:00:35.922765 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5263a15b9461402ebba6425a890e22c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.923254 3071 tablet_bootstrap.cc:654] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.924539 3068 tablet_bootstrap.cc:492] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:35.924638 3068 ts_tablet_manager.cc:1367] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:35.925144 3068 raft_consensus.cc:348] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:35.925259 3068 raft_consensus.cc:374] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.925298 3068 raft_consensus.cc:729] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.925422 3068 consensus_queue.cc:260] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:35.925714 3068 ts_tablet_manager.cc:1398] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.926761 3069 tablet_bootstrap.cc:492] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:35.927747 3069 tablet_bootstrap.cc:654] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.927840 3080 ts_tablet_manager.cc:902] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:35.929212 3071 tablet_bootstrap.cc:492] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:35.929488 3071 ts_tablet_manager.cc:1367] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:35.930145 3071 raft_consensus.cc:348] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.930305 3071 raft_consensus.cc:374] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.930390 3071 raft_consensus.cc:729] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.930550 3071 consensus_queue.cc:260] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.930968 3071 ts_tablet_manager.cc:1398] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.931092 3080 tablet_copy_client.cc:308] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:35.931490 3079 ts_tablet_manager.cc:902] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:35.932153 3079 tablet_copy_client.cc:308] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:35.932307 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 6d3a84c1d004412db381854147330dfc from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:35209)
I20240502 14:00:35.932427 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 6d3a84c1d004412db381854147330dfc from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:35209: session id = d6e2e2c10def4331bec17bd110c9b423-6d3a84c1d004412db381854147330dfc
I20240502 14:00:35.932624 3069 tablet_bootstrap.cc:492] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:35.932729 3069 ts_tablet_manager.cc:1367] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:35.933230 32143 tablet_copy_source_session.cc:205] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:35.933496 3069 raft_consensus.cc:348] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.933614 3069 raft_consensus.cc:374] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.933648 3069 raft_consensus.cc:729] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.933771 3069 consensus_queue.cc:260] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.934093 3069 ts_tablet_manager.cc:1398] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:35.935285 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 6eae206e12b44b809d8b2fcc21611a28 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:49783)
I20240502 14:00:35.935635 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 6eae206e12b44b809d8b2fcc21611a28 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:49783: session id = f0e01c4f581843e9ad4ced97f0072dc4-6eae206e12b44b809d8b2fcc21611a28
I20240502 14:00:35.936303 3080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d3a84c1d004412db381854147330dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.936559 29622 tablet_copy_source_session.cc:205] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:35.938668 3080 tablet_copy_client.cc:783] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:35.939340 3079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae206e12b44b809d8b2fcc21611a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.941529 3079 tablet_copy_client.cc:783] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:35.941660 3079 tablet_copy_client.cc:647] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:35.941731 3049 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:35.942705 3067 tablet_bootstrap.cc:492] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:35.942796 3067 ts_tablet_manager.cc:1367] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.002s
I20240502 14:00:35.944648 3067 raft_consensus.cc:348] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.944851 3067 raft_consensus.cc:374] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.944938 3067 raft_consensus.cc:729] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.945173 3067 consensus_queue.cc:260] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.945878 32233 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.946501 3067 ts_tablet_manager.cc:1398] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:35.946544 3080 tablet_copy_client.cc:647] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:35.947002 3082 tablet_bootstrap.cc:492] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:35.948326 29580 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:35.948544 3049 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:35.948990 3082 tablet_bootstrap.cc:654] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.949256 3049 ts_tablet_manager.cc:1367] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.106s user 0.026s sys 0.004s
I20240502 14:00:35.950196 3049 raft_consensus.cc:348] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.950803 3049 raft_consensus.cc:729] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.951009 32101 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:35.951038 3049 consensus_queue.cc:260] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.952004 3049 ts_tablet_manager.cc:1398] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:35.952852 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-1fc14b4bd42841df91972e0c5d02faa4 received from {username='slave'} at 127.22.55.133:57583
I20240502 14:00:35.952939 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-1fc14b4bd42841df91972e0c5d02faa4 on tablet 1fc14b4bd42841df91972e0c5d02faa4 with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:35.952944 3079 tablet_copy_client.cc:523] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:35.954399 3082 tablet_bootstrap.cc:492] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:35.954507 3079 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:35.954566 3082 ts_tablet_manager.cc:1367] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:35.955086 3082 raft_consensus.cc:348] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.955406 3082 raft_consensus.cc:374] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.955490 3082 raft_consensus.cc:729] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.955686 3082 consensus_queue.cc:260] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.956105 3082 ts_tablet_manager.cc:1398] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.982113 3084 ts_tablet_manager.cc:902] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:35.983068 3084 tablet_copy_client.cc:308] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:35.984021 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet ec8e4ac081604ec8a3f463f740d9119c from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:34291)
I20240502 14:00:35.984144 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet ec8e4ac081604ec8a3f463f740d9119c from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:34291: session id = 8973f2527acb48f5ad36589652de73e1-ec8e4ac081604ec8a3f463f740d9119c
I20240502 14:00:35.984241 3066 tablet_bootstrap.cc:492] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:35.984336 3066 ts_tablet_manager.cc:1367] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.000s
I20240502 14:00:35.984851 3066 raft_consensus.cc:348] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:35.984977 3066 raft_consensus.cc:374] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.985010 3066 raft_consensus.cc:729] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.985145 3066 consensus_queue.cc:260] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:35.985220 32012 tablet_copy_source_session.cc:205] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:35.985453 3066 ts_tablet_manager.cc:1398] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:35.985643 3066 tablet_bootstrap.cc:492] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:35.986521 3066 tablet_bootstrap.cc:654] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.988708 3066 tablet_bootstrap.cc:492] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:35.988806 3066 ts_tablet_manager.cc:1367] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:35.989729 3066 raft_consensus.cc:348] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.989861 3066 raft_consensus.cc:374] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.989934 3066 raft_consensus.cc:729] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.990096 3066 consensus_queue.cc:260] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:35.991223 3066 ts_tablet_manager.cc:1398] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:35.991468 3066 tablet_bootstrap.cc:492] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:35.993764 3066 tablet_bootstrap.cc:654] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:35.994868 3084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8e4ac081604ec8a3f463f740d9119c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:35.997380 3066 tablet_bootstrap.cc:492] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:35.997529 3066 ts_tablet_manager.cc:1367] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 14:00:35.997752 3079 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:35.998317 3066 raft_consensus.cc:348] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.998417 3066 raft_consensus.cc:374] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:35.998453 3066 raft_consensus.cc:729] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:35.998594 3066 consensus_queue.cc:260] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:35.998905 3066 ts_tablet_manager.cc:1398] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:35.999406 3079 tablet_bootstrap.cc:492] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:35.999608 3079 ts_tablet_manager.cc:1367] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.045s user 0.027s sys 0.000s
I20240502 14:00:36.000005 3079 raft_consensus.cc:348] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.000150 3079 raft_consensus.cc:729] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.000311 3079 consensus_queue.cc:260] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.000627 3079 ts_tablet_manager.cc:1398] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.001426 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-6eae206e12b44b809d8b2fcc21611a28 received from {username='slave'} at 127.22.55.133:49783
I20240502 14:00:36.001513 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-6eae206e12b44b809d8b2fcc21611a28 on tablet 6eae206e12b44b809d8b2fcc21611a28 with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:36.004042 3080 tablet_copy_client.cc:523] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.005842 3080 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:36.011158 3080 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.011458 3080 tablet_bootstrap.cc:492] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:36.012430 3084 tablet_copy_client.cc:783] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.013170 3080 ts_tablet_manager.cc:1367] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:00:36.013373 3084 tablet_copy_client.cc:647] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.013712 3080 raft_consensus.cc:348] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:36.013909 3080 raft_consensus.cc:729] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.014245 3080 consensus_queue.cc:260] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:36.014607 3080 ts_tablet_manager.cc:1398] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.016485 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-6d3a84c1d004412db381854147330dfc received from {username='slave'} at 127.22.55.130:35209
I20240502 14:00:36.016577 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-6d3a84c1d004412db381854147330dfc on tablet 6d3a84c1d004412db381854147330dfc with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:36.018046 3084 tablet_copy_client.cc:523] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.019739 3084 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:36.024284 3084 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.024554 3084 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:36.024734 3084 ts_tablet_manager.cc:1367] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 14:00:36.024821 32122 raft_consensus.cc:1203] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.025259 3084 raft_consensus.cc:348] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.025404 3084 raft_consensus.cc:729] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.025589 3084 consensus_queue.cc:260] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.025913 3084 ts_tablet_manager.cc:1398] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:36.026515 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-ec8e4ac081604ec8a3f463f740d9119c received from {username='slave'} at 127.22.55.136:34291
I20240502 14:00:36.026603 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-ec8e4ac081604ec8a3f463f740d9119c on tablet ec8e4ac081604ec8a3f463f740d9119c with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:36.031222 3074 raft_consensus.cc:482] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.031329 3074 raft_consensus.cc:504] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.031774 3074 leader_election.cc:288] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:36.032155 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe66bde51c1d4edfa739f6b9a833b129" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:36.032229 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe66bde51c1d4edfa739f6b9a833b129" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:36.032290 32122 raft_consensus.cc:2454] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:36.032370 29601 raft_consensus.cc:2454] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:36.032543 32188 leader_election.cc:302] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:36.032810 3074 raft_consensus.cc:2781] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.032860 3074 raft_consensus.cc:482] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.032888 3074 raft_consensus.cc:3037] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.033759 3074 raft_consensus.cc:504] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.034089 3074 leader_election.cc:288] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:36.034407 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe66bde51c1d4edfa739f6b9a833b129" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:36.034444 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe66bde51c1d4edfa739f6b9a833b129" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:36.034529 29601 raft_consensus.cc:3037] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.034559 32122 raft_consensus.cc:3037] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.035452 32122 raft_consensus.cc:2454] T fe66bde51c1d4edfa739f6b9a833b129 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:36.035578 29601 raft_consensus.cc:2454] T fe66bde51c1d4edfa739f6b9a833b129 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:36.035713 32188 leader_election.cc:302] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:36.035873 3074 raft_consensus.cc:2781] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.035954 3074 raft_consensus.cc:686] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:36.036094 3074 consensus_queue.cc:237] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.038015 29326 catalog_manager.cc:5484] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:36.067163 31990 raft_consensus.cc:1203] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.071241 858 raft_consensus.cc:482] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.071357 858 raft_consensus.cc:504] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.071816 858 leader_election.cc:288] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:36.072284 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5263a15b9461402ebba6425a890e22c6" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:36.072367 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5263a15b9461402ebba6425a890e22c6" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:36.072436 29470 raft_consensus.cc:2454] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:36.072559 32255 raft_consensus.cc:2454] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:36.072707 31796 leader_election.cc:302] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:36.072885 858 raft_consensus.cc:2781] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.072943 858 raft_consensus.cc:482] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.072981 858 raft_consensus.cc:3037] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.073992 858 raft_consensus.cc:504] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.074354 858 leader_election.cc:288] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:36.074906 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5263a15b9461402ebba6425a890e22c6" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:36.075029 29470 raft_consensus.cc:3037] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.075054 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5263a15b9461402ebba6425a890e22c6" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:36.075479 32255 raft_consensus.cc:3037] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.076025 29470 raft_consensus.cc:2454] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:36.076556 32255 raft_consensus.cc:2454] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:36.078747 31796 leader_election.cc:302] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:36.079349 858 raft_consensus.cc:2781] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.079454 858 raft_consensus.cc:686] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:36.079595 858 consensus_queue.cc:237] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.081748 29329 catalog_manager.cc:5484] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:36.083434 2862 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.083545 2862 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:36.083954 2862 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:36.084997 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bfc5231a542dda0a51aee4543299d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:36.085130 31861 raft_consensus.cc:2454] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:36.085084 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bfc5231a542dda0a51aee4543299d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:36.085227 31990 raft_consensus.cc:2454] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:36.085516 29535 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:36.085701 2862 raft_consensus.cc:2781] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.085769 2862 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.085810 2862 raft_consensus.cc:3037] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.087064 2862 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:36.087414 2862 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:36.087713 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bfc5231a542dda0a51aee4543299d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:36.087831 31990 raft_consensus.cc:3037] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.087795 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bfc5231a542dda0a51aee4543299d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:36.087909 31861 raft_consensus.cc:3037] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.088764 31990 raft_consensus.cc:2454] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:36.088968 31861 raft_consensus.cc:2454] T 8a7bfc5231a542dda0a51aee4543299d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:36.089519 29535 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:36.090512 2862 raft_consensus.cc:2781] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.090600 2862 raft_consensus.cc:686] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:36.090729 2862 consensus_queue.cc:237] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:36.094005 29329 catalog_manager.cc:5484] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:36.140174 858 raft_consensus.cc:482] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.140311 858 raft_consensus.cc:504] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.140724 858 leader_election.cc:288] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:36.141036 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "891851115e2e4d9db5bf3a0013b0edba" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:36.141074 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "891851115e2e4d9db5bf3a0013b0edba" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:36.141181 31990 raft_consensus.cc:2454] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:36.141232 29729 raft_consensus.cc:2454] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:36.141491 31794 leader_election.cc:302] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:36.141690 858 raft_consensus.cc:2781] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.141752 858 raft_consensus.cc:482] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.141785 858 raft_consensus.cc:3037] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.142769 858 raft_consensus.cc:504] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.143162 858 leader_election.cc:288] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:36.143435 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "891851115e2e4d9db5bf3a0013b0edba" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:36.143544 31990 raft_consensus.cc:3037] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.144358 31990 raft_consensus.cc:2454] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:36.144644 31794 leader_election.cc:302] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:36.144627 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "891851115e2e4d9db5bf3a0013b0edba" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:36.144742 29729 raft_consensus.cc:3037] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.144806 858 raft_consensus.cc:2781] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.144903 858 raft_consensus.cc:686] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:36.145033 858 consensus_queue.cc:237] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.145685 29729 raft_consensus.cc:2454] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:36.147701 29327 catalog_manager.cc:5484] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:36.173676 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.178766 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:36.181270 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:36.182612 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:36.183835 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.185360 32101 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:36.188730 31841 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.194321 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:36.196918 32233 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:36.222153 32255 raft_consensus.cc:1261] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:36.222812 858 consensus_queue.cc:1035] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.225493 29470 raft_consensus.cc:1261] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:36.225950 29327 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 5263a15b9461402ebba6425a890e22c6 (table rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998]). Expected version 1 got 0
I20240502 14:00:36.226259 858 consensus_queue.cc:1035] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.230676 29327 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet 5263a15b9461402ebba6425a890e22c6 (table rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998]). Expected version 1 got 0
I20240502 14:00:36.232082 3165 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:36.232501 3165 tablet.cc:1620] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:36.233806 2992 mvcc.cc:205] Tried to move back new op lower bound from 7023240954760605696 to 7023240954184708096. Current Snapshot: MvccSnapshot[applied={T|T < 7023240954760605696}]
I20240502 14:00:36.235427 29327 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet 5263a15b9461402ebba6425a890e22c6 (table rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998]). Expected version 1 got 0
I20240502 14:00:36.237425 3167 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:36.237571 29329 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 5263a15b9461402ebba6425a890e22c6 (table rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998]). Expected version 1 got 0
I20240502 14:00:36.237840 3167 tablet.cc:1620] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:36.239450 3166 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:36.243078 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998] alter complete (version 1)
I20240502 14:00:36.243927 3166 tablet.cc:1620] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:36.346246 29324 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:36.350525 29729 raft_consensus.cc:1203] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:36.359338 32233 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:36.363224 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.367626 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:36.367933 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.368368 3165 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:36.368666 3165 tablet.cc:1620] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:36.373369 3166 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:36.373679 3166 tablet.cc:1620] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:36.376307 29321 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0c29f8825d5c4142b627dae02129f998] alter complete (version 2)
I20240502 14:00:36.376377 31841 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.376899 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:36.380422 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:36.381383 3167 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:36.381706 3167 tablet.cc:1620] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:36.384019 31861 raft_consensus.cc:1203] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.395653 32101 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:36.428874 31861 raft_consensus.cc:1203] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.472807 32254 raft_consensus.cc:1203] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:36.479301 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:36.491992 3074 consensus_queue.cc:1035] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.494701 3165 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:36.495501 3167 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:36.495844 3074 consensus_queue.cc:1035] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.498533 3166 tablet.cc:1722] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:36.503012 29324 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=0c29f8825d5c4142b627dae02129f998] alter complete (version 3)
I20240502 14:00:36.524329 29600 raft_consensus.cc:1203] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:36.557423 2862 consensus_queue.cc:1035] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.561036 2862 consensus_queue.cc:1035] T 8a7bfc5231a542dda0a51aee4543299d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.598131 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.599712 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:36.601903 29844 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:36.602893 32233 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:36.610081 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:36.611017 29321 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:36.611182 29321 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:36.611886 32101 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:36.615182 29321 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 5263a15b9461402ebba6425a890e22c6
I20240502 14:00:36.616065 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5263a15b9461402ebba6425a890e22c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:36 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:36.616235 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 5263a15b9461402ebba6425a890e22c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:36 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:36.618973 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 5263a15b9461402ebba6425a890e22c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:36 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:36.619302 3321 tablet_replica.cc:330] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:36.620066 3321 raft_consensus.cc:2229] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:36.620290 3321 raft_consensus.cc:2258] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:36.625572 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.631309 3321 ts_tablet_manager.cc:1875] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:36.631893 31841 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.634037 3321 ts_tablet_manager.cc:1888] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:36.634222 3321 log.cc:1196] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/5263a15b9461402ebba6425a890e22c6
I20240502 14:00:36.634693 3321 ts_tablet_manager.cc:1909] T 5263a15b9461402ebba6425a890e22c6 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:36.635578 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 5263a15b9461402ebba6425a890e22c6 (table rebalancer_dynamic_table_new_name [id=0c29f8825d5c4142b627dae02129f998]) successfully deleted
I20240502 14:00:36.643736 3320 tablet_replica.cc:330] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:36.643926 3320 raft_consensus.cc:2229] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:36.644062 3320 raft_consensus.cc:2258] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:36.644301 3319 tablet_replica.cc:330] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:36.644479 3319 raft_consensus.cc:2229] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:36.644742 3319 raft_consensus.cc:2258] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:36.644745 3320 ts_tablet_manager.cc:1875] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:36.645332 3319 ts_tablet_manager.cc:1875] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:36.647629 3319 ts_tablet_manager.cc:1888] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:36.647769 3319 log.cc:1196] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/5263a15b9461402ebba6425a890e22c6
I20240502 14:00:36.648082 3319 ts_tablet_manager.cc:1909] T 5263a15b9461402ebba6425a890e22c6 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:36.648865 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 5263a15b9461402ebba6425a890e22c6 (table rebalancer_dynamic_table_new_name [id=0c29f8825d5c4142b627dae02129f998]) successfully deleted
I20240502 14:00:36.652153 3320 ts_tablet_manager.cc:1888] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:36.652099 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:36.658282 3320 log.cc:1196] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/5263a15b9461402ebba6425a890e22c6
I20240502 14:00:36.658720 3320 ts_tablet_manager.cc:1909] T 5263a15b9461402ebba6425a890e22c6 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:36.660185 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8de7efb7024d0290b0e313e7b3d5fd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.660496 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8de7efb7024d0290b0e313e7b3d5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.660447 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 5263a15b9461402ebba6425a890e22c6 (table rebalancer_dynamic_table_new_name [id=0c29f8825d5c4142b627dae02129f998]) successfully deleted
I20240502 14:00:36.661043 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 09f77f3b6bf14a46b2e41c91125f6bd7 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.661083 29449 tablet_service.cc:1467] Processing CreateTablet for tablet c4575ec841b443e3bb0820b604e0f074 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.661319 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f77f3b6bf14a46b2e41c91125f6bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.661379 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4575ec841b443e3bb0820b604e0f074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.662546 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8de7efb7024d0290b0e313e7b3d5fd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.662801 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8de7efb7024d0290b0e313e7b3d5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.663126 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 09f77f3b6bf14a46b2e41c91125f6bd7 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.663401 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f77f3b6bf14a46b2e41c91125f6bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.664726 3324 tablet_bootstrap.cc:492] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:36.664844 32235 tablet_service.cc:1467] Processing CreateTablet for tablet c4575ec841b443e3bb0820b604e0f074 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.665038 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4575ec841b443e3bb0820b604e0f074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.665256 3322 tablet_bootstrap.cc:492] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:36.665704 3324 tablet_bootstrap.cc:654] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.665838 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 09f77f3b6bf14a46b2e41c91125f6bd7 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.666069 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f77f3b6bf14a46b2e41c91125f6bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.666445 3322 tablet_bootstrap.cc:654] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.667349 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 4e8de7efb7024d0290b0e313e7b3d5fd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.667570 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e8de7efb7024d0290b0e313e7b3d5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.668761 3326 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:36.668814 29844 tablet_service.cc:1467] Processing CreateTablet for tablet c4575ec841b443e3bb0820b604e0f074 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.669098 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4575ec841b443e3bb0820b604e0f074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.669649 3326 tablet_bootstrap.cc:654] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.670337 3325 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:36.672000 3325 tablet_bootstrap.cc:654] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.672894 3325 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:36.672984 3325 ts_tablet_manager.cc:1367] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:36.673415 3325 raft_consensus.cc:348] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.673625 3332 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:36.674031 3326 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:36.674787 3332 tablet_bootstrap.cc:654] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.676452 3323 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:36.677479 3323 tablet_bootstrap.cc:654] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.678570 3323 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:36.678656 3323 ts_tablet_manager.cc:1367] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:36.679150 3323 raft_consensus.cc:348] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.679280 3323 raft_consensus.cc:374] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.679316 3323 raft_consensus.cc:729] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.679492 3323 consensus_queue.cc:260] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.679797 3323 ts_tablet_manager.cc:1398] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.683012 3326 ts_tablet_manager.cc:1367] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:36.683625 3326 raft_consensus.cc:348] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.683825 3326 raft_consensus.cc:374] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.683895 3326 raft_consensus.cc:729] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.684084 3326 consensus_queue.cc:260] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.686468 3326 ts_tablet_manager.cc:1398] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:36.681675 1075 consensus_queue.cc:1035] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.686677 3325 raft_consensus.cc:374] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.686733 3325 raft_consensus.cc:729] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.686887 3325 consensus_queue.cc:260] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.688617 3324 tablet_bootstrap.cc:492] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:36.688737 3322 tablet_bootstrap.cc:492] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:36.688745 3324 ts_tablet_manager.cc:1367] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:00:36.688849 3322 ts_tablet_manager.cc:1367] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:00:36.689296 3324 raft_consensus.cc:348] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.689412 3324 raft_consensus.cc:374] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.689442 3324 raft_consensus.cc:729] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.689394 3322 raft_consensus.cc:348] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.689510 3322 raft_consensus.cc:374] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.689550 3322 raft_consensus.cc:729] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.689570 3324 consensus_queue.cc:260] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.689694 3322 consensus_queue.cc:260] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.689910 3324 ts_tablet_manager.cc:1398] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.690013 3322 ts_tablet_manager.cc:1398] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:36.690202 3322 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:36.690384 3324 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:36.691332 3325 ts_tablet_manager.cc:1398] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240502 14:00:36.691380 3324 tablet_bootstrap.cc:654] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.691535 3325 tablet_bootstrap.cc:492] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:36.692785 3325 tablet_bootstrap.cc:654] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.694289 3325 tablet_bootstrap.cc:492] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:36.694388 3325 ts_tablet_manager.cc:1367] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:36.694908 3325 raft_consensus.cc:348] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.695029 3325 raft_consensus.cc:374] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.695062 3325 raft_consensus.cc:729] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.695181 3325 consensus_queue.cc:260] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.695514 3325 ts_tablet_manager.cc:1398] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.696714 858 consensus_queue.cc:1035] T 891851115e2e4d9db5bf3a0013b0edba P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:36.698931 3324 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:36.699024 3324 ts_tablet_manager.cc:1367] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:36.699576 3324 raft_consensus.cc:348] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.699721 3324 raft_consensus.cc:374] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.699751 3324 raft_consensus.cc:729] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.699872 3324 consensus_queue.cc:260] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.701021 3324 ts_tablet_manager.cc:1398] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:36.707001 2821 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.707114 2821 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.707516 3322 tablet_bootstrap.cc:654] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.707544 2821 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:36.708611 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:36.708745 31861 raft_consensus.cc:2454] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:36.709373 3332 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:36.709501 3332 ts_tablet_manager.cc:1367] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240502 14:00:36.709687 32056 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:36.709879 2821 raft_consensus.cc:2781] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.709929 2821 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.709966 2821 raft_consensus.cc:3037] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.710201 3332 raft_consensus.cc:348] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.710361 3332 raft_consensus.cc:374] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.710426 3332 raft_consensus.cc:729] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.710578 3332 consensus_queue.cc:260] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.710943 2821 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.711263 2821 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:36.711009 3332 ts_tablet_manager.cc:1398] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.711745 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:36.711863 31861 raft_consensus.cc:3037] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.711804 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:36.711804 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
W20240502 14:00:36.712397 32056 leader_election.cc:341] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:00:36.712502 32056 leader_election.cc:341] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:36.712813 31861 raft_consensus.cc:2454] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:36.713133 32056 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:36.713383 2821 raft_consensus.cc:2781] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.713452 2821 raft_consensus.cc:686] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:36.713580 2821 consensus_queue.cc:237] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.715626 29323 catalog_manager.cc:5484] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:36.716710 29329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:36.723923 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 5fba92ae1b924c64ae1f6761c79a7b8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7a2883e39ca4902920a835aacf94fce]), partition=
I20240502 14:00:36.724225 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fba92ae1b924c64ae1f6761c79a7b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.725033 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 5fba92ae1b924c64ae1f6761c79a7b8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7a2883e39ca4902920a835aacf94fce]), partition=
I20240502 14:00:36.725255 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fba92ae1b924c64ae1f6761c79a7b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.727011 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 5fba92ae1b924c64ae1f6761c79a7b8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7a2883e39ca4902920a835aacf94fce]), partition=
I20240502 14:00:36.727262 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fba92ae1b924c64ae1f6761c79a7b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.727463 3326 tablet_bootstrap.cc:492] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:36.728888 3322 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:36.728996 3322 ts_tablet_manager.cc:1367] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240502 14:00:36.729490 3323 tablet_bootstrap.cc:492] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:36.729525 3322 raft_consensus.cc:348] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.729655 3322 raft_consensus.cc:374] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.729696 3322 raft_consensus.cc:729] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.729825 3322 consensus_queue.cc:260] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.730165 3322 ts_tablet_manager.cc:1398] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.730563 3323 tablet_bootstrap.cc:654] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.731438 3325 tablet_bootstrap.cc:492] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:36.732264 3325 tablet_bootstrap.cc:654] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.733907 3326 tablet_bootstrap.cc:654] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.735235 3325 tablet_bootstrap.cc:492] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:36.735312 3325 ts_tablet_manager.cc:1367] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:36.735723 3325 raft_consensus.cc:348] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.735824 3325 raft_consensus.cc:374] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.735852 3325 raft_consensus.cc:729] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.735954 3325 consensus_queue.cc:260] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.736184 3325 ts_tablet_manager.cc:1398] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.736929 3323 tablet_bootstrap.cc:492] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:36.737005 3323 ts_tablet_manager.cc:1367] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:36.737471 3323 raft_consensus.cc:348] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.737581 3323 raft_consensus.cc:374] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.737614 3323 raft_consensus.cc:729] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.737721 3323 consensus_queue.cc:260] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.737957 3323 ts_tablet_manager.cc:1398] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.739460 3326 tablet_bootstrap.cc:492] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:36.739533 3326 ts_tablet_manager.cc:1367] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:36.739930 3326 raft_consensus.cc:348] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.740015 3326 raft_consensus.cc:374] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.740041 3326 raft_consensus.cc:729] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.740149 3326 consensus_queue.cc:260] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.740392 3326 ts_tablet_manager.cc:1398] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.749557 3215 raft_consensus.cc:482] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.749665 3215 raft_consensus.cc:504] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.750097 3215 leader_election.cc:288] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:36.750689 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4575ec841b443e3bb0820b604e0f074" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:36.750833 32254 raft_consensus.cc:2454] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:36.751188 29799 leader_election.cc:302] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:36.751446 3215 raft_consensus.cc:2781] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.751411 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4575ec841b443e3bb0820b604e0f074" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:36.751505 3215 raft_consensus.cc:482] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.751549 3215 raft_consensus.cc:3037] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.751542 29470 raft_consensus.cc:2454] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:36.752557 3215 raft_consensus.cc:504] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.752861 3215 leader_election.cc:288] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:36.753235 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4575ec841b443e3bb0820b604e0f074" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:36.753249 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4575ec841b443e3bb0820b604e0f074" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:36.753360 32255 raft_consensus.cc:3037] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.753382 29470 raft_consensus.cc:3037] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.754387 32255 raft_consensus.cc:2454] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:36.754398 29470 raft_consensus.cc:2454] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:36.754724 29799 leader_election.cc:302] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:36.755026 3215 raft_consensus.cc:2781] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.755139 3215 raft_consensus.cc:686] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:36.755268 3215 consensus_queue.cc:237] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:36.757686 29329 catalog_manager.cc:5484] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:36.758841 29601 consensus_queue.cc:237] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:36.760017 29863 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.760128 31992 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.760381 1537 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.760526 2862 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.762173 3364 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:36.762552 29863 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:36.762594 31992 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:36.765008 29329 catalog_manager.cc:5484] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:36.767681 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2ab1ded2c241038d5685678aa819a8 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:36.768685 3365 tablet_replica.cc:330] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:36.768857 3365 raft_consensus.cc:2229] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:36.769065 3365 raft_consensus.cc:2258] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:36.769600 3365 ts_tablet_manager.cc:1875] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:36.771957 3365 ts_tablet_manager.cc:1888] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:36.772049 3365 log.cc:1196] T 8d2ab1ded2c241038d5685678aa819a8 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/8d2ab1ded2c241038d5685678aa819a8
I20240502 14:00:36.772620 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 8d2ab1ded2c241038d5685678aa819a8 (table rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]) successfully deleted
I20240502 14:00:36.863493 2680 raft_consensus.cc:482] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.863619 2680 raft_consensus.cc:504] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.863972 2680 leader_election.cc:288] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:36.864341 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8de7efb7024d0290b0e313e7b3d5fd" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:36.864384 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8de7efb7024d0290b0e313e7b3d5fd" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:36.864475 31992 raft_consensus.cc:2454] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:36.864512 31861 raft_consensus.cc:2454] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:36.864770 32188 leader_election.cc:302] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:36.864933 2680 raft_consensus.cc:2781] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.864995 2680 raft_consensus.cc:482] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.865032 2680 raft_consensus.cc:3037] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.866200 2680 raft_consensus.cc:504] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.866535 2680 leader_election.cc:288] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:36.866822 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8de7efb7024d0290b0e313e7b3d5fd" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:36.866942 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e8de7efb7024d0290b0e313e7b3d5fd" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:36.866990 31992 raft_consensus.cc:3037] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.867070 31861 raft_consensus.cc:3037] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.867811 31992 raft_consensus.cc:2454] T 4e8de7efb7024d0290b0e313e7b3d5fd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:36.867836 31861 raft_consensus.cc:2454] T 4e8de7efb7024d0290b0e313e7b3d5fd P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:36.868059 32188 leader_election.cc:302] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:36.868228 2680 raft_consensus.cc:2781] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.868315 2680 raft_consensus.cc:686] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:36.868477 2680 consensus_queue.cc:237] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:36.870380 29323 catalog_manager.cc:5484] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:36.921880 3239 raft_consensus.cc:482] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.922036 3239 raft_consensus.cc:504] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.922474 3239 leader_election.cc:288] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:36.922837 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba92ae1b924c64ae1f6761c79a7b8c" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:36.922996 32119 raft_consensus.cc:2454] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:36.922933 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba92ae1b924c64ae1f6761c79a7b8c" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:36.923105 31861 raft_consensus.cc:2454] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:36.923316 29404 leader_election.cc:302] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:36.923507 3239 raft_consensus.cc:2781] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.923575 3239 raft_consensus.cc:482] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.923625 3239 raft_consensus.cc:3037] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.924760 3239 raft_consensus.cc:504] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.925103 3239 leader_election.cc:288] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:36.925390 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba92ae1b924c64ae1f6761c79a7b8c" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:36.925455 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fba92ae1b924c64ae1f6761c79a7b8c" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:36.925500 32119 raft_consensus.cc:3037] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.925571 31861 raft_consensus.cc:3037] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.926321 32119 raft_consensus.cc:2454] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:36.926432 31861 raft_consensus.cc:2454] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:36.926615 29404 leader_election.cc:302] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:36.926821 3239 raft_consensus.cc:2781] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.926946 3239 raft_consensus.cc:686] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:36.927114 3239 consensus_queue.cc:237] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:36.929543 29323 catalog_manager.cc:5484] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:36.952957 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:36.957801 31972 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:36.958045 32235 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.959789 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:36.962927 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:36.975649 32101 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:36.987587 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.990926 31841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:37.053102 29863 consensus_queue.cc:237] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.054409 32119 raft_consensus.cc:1261] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.054622 29729 raft_consensus.cc:1261] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.054823 2740 consensus_queue.cc:1035] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.055085 3215 consensus_queue.cc:1035] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.056897 3439 raft_consensus.cc:2932] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:37.057875 32119 raft_consensus.cc:2932] T 59f4faff46704ca1837edd139596d9e2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:37.060015 29729 raft_consensus.cc:2932] T 59f4faff46704ca1837edd139596d9e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:37.060407 29323 catalog_manager.cc:5484] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:37.064975 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f4faff46704ca1837edd139596d9e2 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:37.065867 3445 tablet_replica.cc:330] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:37.066069 3445 raft_consensus.cc:2229] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.066967 3445 raft_consensus.cc:2258] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.067544 3445 ts_tablet_manager.cc:1875] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.070040 3445 ts_tablet_manager.cc:1888] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:37.070119 3445 log.cc:1196] T 59f4faff46704ca1837edd139596d9e2 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/59f4faff46704ca1837edd139596d9e2
I20240502 14:00:37.070668 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 59f4faff46704ca1837edd139596d9e2 (table rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]) successfully deleted
I20240502 14:00:37.072201 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:37.076552 32119 raft_consensus.cc:1261] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.076647 31861 raft_consensus.cc:1261] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.077018 1767 consensus_queue.cc:1035] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.077313 1767 consensus_queue.cc:1035] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.079553 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 5fba92ae1b924c64ae1f6761c79a7b8c (table rebalancer_dynamic_table [id=b7a2883e39ca4902920a835aacf94fce]). Expected version 1 got 0
I20240502 14:00:37.080844 3457 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:37.081254 3457 tablet.cc:1620] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:37.081763 3458 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:37.082156 3458 tablet.cc:1620] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:37.083498 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 5fba92ae1b924c64ae1f6761c79a7b8c (table rebalancer_dynamic_table [id=b7a2883e39ca4902920a835aacf94fce]). Expected version 1 got 0
I20240502 14:00:37.084702 3456 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:37.084985 3456 tablet.cc:1620] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:37.085693 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=b7a2883e39ca4902920a835aacf94fce] alter complete (version 1)
I20240502 14:00:37.135783 31992 raft_consensus.cc:3037] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.137859 2231 consensus_queue.cc:1035] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.143095 3329 consensus_queue.cc:1035] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.165107 3439 consensus_queue.cc:1035] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.172286 2740 consensus_queue.cc:1035] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.173163 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:37.177425 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:37.180090 31841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:37.181738 32235 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:37.189325 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:37.190829 29713 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:37.199445 3456 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:37.199774 3456 tablet.cc:1620] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:37.200121 3457 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:37.200382 3457 tablet.cc:1620] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:37.203501 3458 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:37.203826 3458 tablet.cc:1620] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:37.206579 31972 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:37.209044 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:37.209352 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=b7a2883e39ca4902920a835aacf94fce] alter complete (version 2)
I20240502 14:00:37.223137 32101 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:37.279167 29600 consensus_queue.cc:237] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:37.280405 29863 raft_consensus.cc:1261] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:37.280447 31992 raft_consensus.cc:1261] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:37.280781 3364 consensus_queue.cc:1035] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:37.280974 2862 consensus_queue.cc:1035] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:37.282689 3526 raft_consensus.cc:2932] T e9f48a525d8c4a0783f0450aa6ec7cb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:37.283051 29863 raft_consensus.cc:2932] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:37.283077 31992 raft_consensus.cc:2932] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:37.286578 29324 catalog_manager.cc:5484] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:37.290058 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f48a525d8c4a0783f0450aa6ec7cb0 with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:37.290433 3531 tablet_replica.cc:330] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:37.290580 3531 raft_consensus.cc:2229] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.290700 3531 raft_consensus.cc:2258] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.292222 3531 ts_tablet_manager.cc:1875] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.294770 3531 ts_tablet_manager.cc:1888] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:37.294872 3531 log.cc:1196] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/e9f48a525d8c4a0783f0450aa6ec7cb0
I20240502 14:00:37.295528 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet e9f48a525d8c4a0783f0450aa6ec7cb0 (table rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]) successfully deleted
I20240502 14:00:37.321553 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:37.328675 3456 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:37.330248 3457 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:37.331619 3458 tablet.cc:1722] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:37.335506 29321 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=b7a2883e39ca4902920a835aacf94fce] alter complete (version 3)
I20240502 14:00:37.350917 2680 consensus_queue.cc:1035] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.357108 3074 consensus_queue.cc:1035] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.383728 29329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:37.391784 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 25d043c246e94f3090295583133ca0cd (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.391860 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 13d5181b92f946b99c26735bb05b5974 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.392123 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d043c246e94f3090295583133ca0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.392123 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d5181b92f946b99c26735bb05b5974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.392078 29451 tablet_service.cc:1467] Processing CreateTablet for tablet f57fc57c01324259ba1bca6d049e018e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.392263 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 25d043c246e94f3090295583133ca0cd (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.392406 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57fc57c01324259ba1bca6d049e018e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.392525 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d043c246e94f3090295583133ca0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.394906 32233 tablet_service.cc:1467] Processing CreateTablet for tablet f57fc57c01324259ba1bca6d049e018e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.395156 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57fc57c01324259ba1bca6d049e018e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.396021 3572 tablet_bootstrap.cc:492] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:37.394829 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 25d043c246e94f3090295583133ca0cd (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.396895 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d043c246e94f3090295583133ca0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.396927 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 13d5181b92f946b99c26735bb05b5974 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.397154 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d5181b92f946b99c26735bb05b5974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.397315 3572 tablet_bootstrap.cc:654] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.397557 31972 tablet_service.cc:1467] Processing CreateTablet for tablet f57fc57c01324259ba1bca6d049e018e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.397816 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57fc57c01324259ba1bca6d049e018e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.398187 3571 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:37.399130 3571 tablet_bootstrap.cc:654] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.395879 3573 tablet_bootstrap.cc:492] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:37.400218 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 13d5181b92f946b99c26735bb05b5974 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.400290 3574 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:37.400424 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d5181b92f946b99c26735bb05b5974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.401016 3573 tablet_bootstrap.cc:654] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.401127 3574 tablet_bootstrap.cc:654] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.402449 3578 tablet_bootstrap.cc:492] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:37.402560 3572 tablet_bootstrap.cc:492] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:37.402624 3572 ts_tablet_manager.cc:1367] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:37.403120 3572 raft_consensus.cc:348] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.403223 3572 raft_consensus.cc:374] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.403244 3572 raft_consensus.cc:729] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.403357 3572 consensus_queue.cc:260] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.403475 3578 tablet_bootstrap.cc:654] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.403659 3572 ts_tablet_manager.cc:1398] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:37.404291 3573 tablet_bootstrap.cc:492] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:37.404451 3573 ts_tablet_manager.cc:1367] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:37.404947 3571 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:37.405157 3571 ts_tablet_manager.cc:1367] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:37.405292 3573 raft_consensus.cc:348] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.405481 3573 raft_consensus.cc:374] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.405557 3573 raft_consensus.cc:729] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.405758 3571 raft_consensus.cc:348] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.406838 858 raft_consensus.cc:482] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.406945 858 raft_consensus.cc:504] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.407078 3571 raft_consensus.cc:374] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.407138 3571 raft_consensus.cc:729] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.407346 858 leader_election.cc:288] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:37.407346 3571 consensus_queue.cc:260] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.405742 3573 consensus_queue.cc:260] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.407611 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d5181b92f946b99c26735bb05b5974" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:37.407835 3571 ts_tablet_manager.cc:1398] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:37.407850 3573 ts_tablet_manager.cc:1398] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:37.407999 3571 tablet_bootstrap.cc:492] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:37.408073 3573 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:37.408910 3571 tablet_bootstrap.cc:654] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.409348 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d5181b92f946b99c26735bb05b5974" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
W20240502 14:00:37.409794 31795 leader_election.cc:341] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:00:37.410094 31794 leader_election.cc:341] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:37.410156 31794 leader_election.cc:302] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:37.410539 1075 raft_consensus.cc:2726] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:00:37.412274 3574 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:37.412456 3574 ts_tablet_manager.cc:1367] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:37.413082 3574 raft_consensus.cc:348] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.413251 3574 raft_consensus.cc:374] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.413336 3574 raft_consensus.cc:729] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.413565 3574 consensus_queue.cc:260] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.413746 3571 tablet_bootstrap.cc:492] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:37.413841 3571 ts_tablet_manager.cc:1367] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:37.414041 3574 ts_tablet_manager.cc:1398] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.414134 3573 tablet_bootstrap.cc:654] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.414366 3571 raft_consensus.cc:348] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.414469 3571 raft_consensus.cc:374] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.414499 3571 raft_consensus.cc:729] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.414603 3571 consensus_queue.cc:260] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.414691 2777 raft_consensus.cc:482] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.414770 2777 raft_consensus.cc:504] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.414907 3571 ts_tablet_manager.cc:1398] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.415148 2777 leader_election.cc:288] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:37.415762 3574 tablet_bootstrap.cc:492] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:37.415776 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d043c246e94f3090295583133ca0cd" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:37.416178 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d043c246e94f3090295583133ca0cd" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:37.407254 3578 tablet_bootstrap.cc:492] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:37.416311 32255 raft_consensus.cc:2454] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:37.416339 3578 ts_tablet_manager.cc:1367] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240502 14:00:37.416795 3574 tablet_bootstrap.cc:654] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.416852 3578 raft_consensus.cc:348] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.416962 3578 raft_consensus.cc:374] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.417003 3578 raft_consensus.cc:729] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.417119 3578 consensus_queue.cc:260] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
W20240502 14:00:37.417515 29668 leader_election.cc:341] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:37.417591 3578 ts_tablet_manager.cc:1398] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.417789 3578 tablet_bootstrap.cc:492] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:37.418339 3573 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:37.418429 3573 ts_tablet_manager.cc:1367] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:37.418583 29668 leader_election.cc:302] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:37.418759 3578 tablet_bootstrap.cc:654] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.418964 3573 raft_consensus.cc:348] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.419071 3573 raft_consensus.cc:374] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.419109 3573 raft_consensus.cc:729] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.419231 3573 consensus_queue.cc:260] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.419526 3573 ts_tablet_manager.cc:1398] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:37.421087 3584 raft_consensus.cc:2781] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.421276 3584 raft_consensus.cc:482] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.421375 3584 raft_consensus.cc:3037] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.422930 3584 raft_consensus.cc:504] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.423935 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d043c246e94f3090295583133ca0cd" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:37.424059 29470 raft_consensus.cc:3037] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.424988 29470 raft_consensus.cc:2454] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:37.425266 29668 leader_election.cc:302] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:37.425459 2777 raft_consensus.cc:2781] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.425659 2777 raft_consensus.cc:686] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:37.425801 2777 consensus_queue.cc:237] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.427893 29325 catalog_manager.cc:5484] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:37.428483 3584 leader_election.cc:288] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:37.428668 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d043c246e94f3090295583133ca0cd" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:37.428789 32255 raft_consensus.cc:3037] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.429805 32255 raft_consensus.cc:2454] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:37.432056 3578 tablet_bootstrap.cc:492] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:37.432191 3578 ts_tablet_manager.cc:1367] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:37.432729 3578 raft_consensus.cc:348] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.432848 3578 raft_consensus.cc:374] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.432883 3578 raft_consensus.cc:729] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.433015 3578 consensus_queue.cc:260] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.433321 3578 ts_tablet_manager.cc:1398] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.434091 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:37.437175 29324 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:37.437573 29324 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:37.440937 31971 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:37.441428 29324 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 5fba92ae1b924c64ae1f6761c79a7b8c
I20240502 14:00:37.441840 3574 tablet_bootstrap.cc:492] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:37.441946 3574 ts_tablet_manager.cc:1367] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240502 14:00:37.442616 3574 raft_consensus.cc:348] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.442728 3574 raft_consensus.cc:374] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.442715 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fba92ae1b924c64ae1f6761c79a7b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:37 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:37.442764 3574 raft_consensus.cc:729] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.442972 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fba92ae1b924c64ae1f6761c79a7b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:37 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:37.443094 3601 tablet_replica.cc:330] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:37.443333 3602 tablet_replica.cc:330] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:37.443425 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fba92ae1b924c64ae1f6761c79a7b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:37 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:37.443490 3602 raft_consensus.cc:2229] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:37.443711 3602 raft_consensus.cc:2258] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.443742 3601 raft_consensus.cc:2229] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.444236 3601 raft_consensus.cc:2258] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.444720 3602 ts_tablet_manager.cc:1875] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:37.445351 3601 ts_tablet_manager.cc:1875] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:37.445510 3574 consensus_queue.cc:260] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.445845 3574 ts_tablet_manager.cc:1398] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:37.449664 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:37.450119 3602 ts_tablet_manager.cc:1888] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:37.450191 3602 log.cc:1196] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/5fba92ae1b924c64ae1f6761c79a7b8c
I20240502 14:00:37.450762 3602 ts_tablet_manager.cc:1909] T 5fba92ae1b924c64ae1f6761c79a7b8c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:37.451139 31841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:37.451519 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 5fba92ae1b924c64ae1f6761c79a7b8c (table rebalancer_dynamic_table_new_name [id=b7a2883e39ca4902920a835aacf94fce]) successfully deleted
I20240502 14:00:37.452699 3601 ts_tablet_manager.cc:1888] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:37.452772 3601 log.cc:1196] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/5fba92ae1b924c64ae1f6761c79a7b8c
I20240502 14:00:37.453222 3601 ts_tablet_manager.cc:1909] T 5fba92ae1b924c64ae1f6761c79a7b8c P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:37.453891 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 5fba92ae1b924c64ae1f6761c79a7b8c (table rebalancer_dynamic_table_new_name [id=b7a2883e39ca4902920a835aacf94fce]) successfully deleted
I20240502 14:00:37.454432 3603 tablet_replica.cc:330] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:37.454563 3603 raft_consensus.cc:2229] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.454664 3603 raft_consensus.cc:2258] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.455773 2680 raft_consensus.cc:482] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.455888 3603 ts_tablet_manager.cc:1875] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:37.455866 2680 raft_consensus.cc:504] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.455935 32233 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:37.456279 2680 leader_election.cc:288] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:37.456740 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fc57c01324259ba1bca6d049e018e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:37.456874 31990 raft_consensus.cc:2454] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:37.457279 32188 leader_election.cc:302] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:37.457695 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fc57c01324259ba1bca6d049e018e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:37.457830 29470 raft_consensus.cc:2454] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:37.458240 2680 raft_consensus.cc:2781] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.458304 2680 raft_consensus.cc:482] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.458353 2680 raft_consensus.cc:3037] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.459116 3603 ts_tablet_manager.cc:1888] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:37.459261 3603 log.cc:1196] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/5fba92ae1b924c64ae1f6761c79a7b8c
I20240502 14:00:37.459611 3603 ts_tablet_manager.cc:1909] T 5fba92ae1b924c64ae1f6761c79a7b8c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:37.461001 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:37.461390 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 5fba92ae1b924c64ae1f6761c79a7b8c (table rebalancer_dynamic_table_new_name [id=b7a2883e39ca4902920a835aacf94fce]) successfully deleted
I20240502 14:00:37.462131 2680 raft_consensus.cc:504] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.462495 2680 leader_election.cc:288] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:37.462829 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fc57c01324259ba1bca6d049e018e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:37.462980 31990 raft_consensus.cc:3037] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.463676 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:37.463776 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57fc57c01324259ba1bca6d049e018e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:37.463894 29470 raft_consensus.cc:3037] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.464062 31990 raft_consensus.cc:2454] T f57fc57c01324259ba1bca6d049e018e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:37.464458 32188 leader_election.cc:302] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:37.464921 29470 raft_consensus.cc:2454] T f57fc57c01324259ba1bca6d049e018e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:37.465471 3074 raft_consensus.cc:2781] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.465549 3074 raft_consensus.cc:686] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:37.465667 3074 consensus_queue.cc:237] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:37.468076 29320 catalog_manager.cc:5484] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:37.470461 32101 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:37.544059 31861 consensus_queue.cc:237] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.545542 29729 raft_consensus.cc:1261] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:37.545681 32255 raft_consensus.cc:1261] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:37.546067 858 consensus_queue.cc:1035] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:37.546247 1075 consensus_queue.cc:1035] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:37.546427 29320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:37.549477 858 raft_consensus.cc:2932] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:37.552225 32255 raft_consensus.cc:2932] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:37.553740 29320 catalog_manager.cc:5484] T 27e9d8ef4f754df3817cc4a981549e02 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:37.555424 31841 tablet_service.cc:1467] Processing CreateTablet for tablet ecb724afc4d545f8a709d34c8d5b72be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76]), partition=
I20240502 14:00:37.555704 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb724afc4d545f8a709d34c8d5b72be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.555871 32233 tablet_service.cc:1467] Processing CreateTablet for tablet ecb724afc4d545f8a709d34c8d5b72be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76]), partition=
I20240502 14:00:37.556069 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb724afc4d545f8a709d34c8d5b72be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.557873 3572 tablet_bootstrap.cc:492] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:37.558051 3574 tablet_bootstrap.cc:492] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:37.558107 32101 tablet_service.cc:1467] Processing CreateTablet for tablet ecb724afc4d545f8a709d34c8d5b72be (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76]), partition=
I20240502 14:00:37.558363 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb724afc4d545f8a709d34c8d5b72be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.558818 3572 tablet_bootstrap.cc:654] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.558907 3574 tablet_bootstrap.cc:654] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.559803 29729 raft_consensus.cc:2932] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:37.560245 3574 tablet_bootstrap.cc:492] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:37.560315 3574 ts_tablet_manager.cc:1367] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.560775 3574 raft_consensus.cc:348] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.560910 3574 raft_consensus.cc:374] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.560950 3574 raft_consensus.cc:729] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.561069 3574 consensus_queue.cc:260] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.561254 3572 tablet_bootstrap.cc:492] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:37.561336 3572 ts_tablet_manager.cc:1367] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:37.561383 3574 ts_tablet_manager.cc:1398] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.561797 3572 raft_consensus.cc:348] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.561908 3572 raft_consensus.cc:374] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.561934 3572 raft_consensus.cc:729] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.562043 3572 consensus_queue.cc:260] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.562294 3572 ts_tablet_manager.cc:1398] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.563434 3625 tablet_bootstrap.cc:492] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:37.564491 3625 tablet_bootstrap.cc:654] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.565796 3625 tablet_bootstrap.cc:492] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:37.565912 3625 ts_tablet_manager.cc:1367] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:37.567281 3625 raft_consensus.cc:348] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.567487 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e9d8ef4f754df3817cc4a981549e02 with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:37.567564 3625 raft_consensus.cc:374] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.567611 3625 raft_consensus.cc:729] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.567739 3625 consensus_queue.cc:260] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.568135 3625 ts_tablet_manager.cc:1398] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.567927 3627 tablet_replica.cc:330] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:37.568646 3627 raft_consensus.cc:2229] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.568784 3627 raft_consensus.cc:2258] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.571861 3627 ts_tablet_manager.cc:1875] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.574576 3627 ts_tablet_manager.cc:1888] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:37.574661 3627 log.cc:1196] T 27e9d8ef4f754df3817cc4a981549e02 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/27e9d8ef4f754df3817cc4a981549e02
I20240502 14:00:37.575196 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 27e9d8ef4f754df3817cc4a981549e02 (table rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]) successfully deleted
I20240502 14:00:37.686476 2680 raft_consensus.cc:482] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.686599 2680 raft_consensus.cc:504] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.687008 2680 leader_election.cc:288] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:37.687342 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb724afc4d545f8a709d34c8d5b72be" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:37.687413 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb724afc4d545f8a709d34c8d5b72be" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:37.687480 31861 raft_consensus.cc:2454] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:37.687582 32120 raft_consensus.cc:2454] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:37.687789 32188 leader_election.cc:302] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:37.687942 2680 raft_consensus.cc:2781] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.688004 2680 raft_consensus.cc:482] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.688040 2680 raft_consensus.cc:3037] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.689131 2680 raft_consensus.cc:504] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.689422 2680 leader_election.cc:288] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:37.689747 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb724afc4d545f8a709d34c8d5b72be" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:37.689811 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb724afc4d545f8a709d34c8d5b72be" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:37.689888 31861 raft_consensus.cc:3037] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.689952 32120 raft_consensus.cc:3037] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.690726 31861 raft_consensus.cc:2454] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:37.690948 32120 raft_consensus.cc:2454] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:37.691022 32188 leader_election.cc:302] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:37.691184 2680 raft_consensus.cc:2781] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.691273 2680 raft_consensus.cc:686] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:37.691414 2680 consensus_queue.cc:237] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:37.693437 29322 catalog_manager.cc:5484] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:37.745980 3575 raft_consensus.cc:482] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.746101 3575 raft_consensus.cc:504] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.746618 3575 leader_election.cc:288] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:37.746946 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d5181b92f946b99c26735bb05b5974" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:37.746989 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d5181b92f946b99c26735bb05b5974" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:37.747085 29729 raft_consensus.cc:2454] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:37.747120 31861 raft_consensus.cc:2454] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:37.747452 31925 leader_election.cc:302] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:37.748016 3575 raft_consensus.cc:2781] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.748085 3575 raft_consensus.cc:482] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.748124 3575 raft_consensus.cc:3037] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.749076 3575 raft_consensus.cc:504] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.749377 3575 leader_election.cc:288] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:37.749668 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d5181b92f946b99c26735bb05b5974" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:37.749703 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d5181b92f946b99c26735bb05b5974" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:37.749774 29729 raft_consensus.cc:3037] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.749830 31861 raft_consensus.cc:3037] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.750739 29729 raft_consensus.cc:2454] T 13d5181b92f946b99c26735bb05b5974 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:37.750825 31861 raft_consensus.cc:2454] T 13d5181b92f946b99c26735bb05b5974 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:37.751073 31926 leader_election.cc:302] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:37.751346 3575 raft_consensus.cc:2781] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.751421 3575 raft_consensus.cc:686] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:37.751561 3575 consensus_queue.cc:237] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:37.753289 29322 catalog_manager.cc:5484] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:37.777808 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:37.778311 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:37.780342 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:37.786599 32101 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:37.788285 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:37.791025 31841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:37.795197 32233 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:37.796449 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:37.819780 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:37.824479 32120 raft_consensus.cc:1261] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.825181 2680 consensus_queue.cc:1035] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.825726 31861 raft_consensus.cc:1261] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.827800 29322 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet ecb724afc4d545f8a709d34c8d5b72be (table rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76]). Expected version 1 got 0
I20240502 14:00:37.829142 2680 consensus_queue.cc:1035] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.829833 3458 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:37.830173 3458 tablet.cc:1620] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:37.832548 29322 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet ecb724afc4d545f8a709d34c8d5b72be (table rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76]). Expected version 1 got 0
I20240502 14:00:37.833159 3161 mvcc.cc:205] Tried to move back new op lower bound from 7023240961324716032 to 7023240960785387520. Current Snapshot: MvccSnapshot[applied={T|T < 7023240961324716032}]
I20240502 14:00:37.834069 29322 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet ecb724afc4d545f8a709d34c8d5b72be (table rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76]). Expected version 1 got 0
I20240502 14:00:37.844393 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76] alter complete (version 1)
I20240502 14:00:37.845013 3686 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:37.845379 3686 tablet.cc:1620] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:37.846279 3689 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:37.849382 3689 tablet.cc:1620] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:37.851312 2680 consensus_queue.cc:1035] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.854944 3074 consensus_queue.cc:1035] T f57fc57c01324259ba1bca6d049e018e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.877679 29470 consensus_queue.cc:237] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:37.879005 31861 raft_consensus.cc:1261] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.879403 29732 raft_consensus.cc:1261] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.879400 3239 consensus_queue.cc:1035] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.879861 3239 consensus_queue.cc:1035] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.881515 3507 raft_consensus.cc:2932] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:37.881909 31861 raft_consensus.cc:2932] T aeaf1db0298741fa813660a8e38d01f9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:37.885166 29322 catalog_manager.cc:5484] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:37.888955 29732 raft_consensus.cc:2932] T aeaf1db0298741fa813660a8e38d01f9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:37.889705 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet aeaf1db0298741fa813660a8e38d01f9 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:37.893244 3717 tablet_replica.cc:330] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:37.893514 3717 raft_consensus.cc:2229] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.893719 3717 raft_consensus.cc:2258] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.895087 3717 ts_tablet_manager.cc:1875] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.897687 3717 ts_tablet_manager.cc:1888] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:37.897754 3717 log.cc:1196] T aeaf1db0298741fa813660a8e38d01f9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/aeaf1db0298741fa813660a8e38d01f9
I20240502 14:00:37.898309 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet aeaf1db0298741fa813660a8e38d01f9 (table rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]) successfully deleted
I20240502 14:00:37.930356 3599 consensus_queue.cc:1035] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.934298 3599 consensus_queue.cc:1035] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.947415 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:37.953116 3686 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:37.953495 3686 tablet.cc:1620] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:37.955662 3458 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:37.955979 3458 tablet.cc:1620] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:37.956274 3689 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:37.956545 3689 tablet.cc:1620] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:37.966146 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=7b3b343cd3234696bb95fb0979489b76] alter complete (version 2)
I20240502 14:00:38.012717 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.013221 31971 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:38.014899 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:38.017285 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:38.020164 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:38.025425 32233 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:38.038192 32101 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:38.057511 31841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:38.071336 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:38.077595 3686 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:38.078410 3458 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:38.079331 3689 tablet.cc:1722] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:38.085461 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=7b3b343cd3234696bb95fb0979489b76] alter complete (version 3)
I20240502 14:00:38.109364 31861 consensus_queue.cc:237] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:38.110579 29863 raft_consensus.cc:1261] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:00:38.110700 32120 raft_consensus.cc:1261] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:00:38.111168 858 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.111378 1075 consensus_queue.cc:1035] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.113392 1075 raft_consensus.cc:2932] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:38.114928 32120 raft_consensus.cc:2932] T fe523f0c364f447cb2a62b599d030cca P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:38.115446 29863 raft_consensus.cc:2932] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:38.117050 29320 catalog_manager.cc:5484] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 2 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 2 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:38.120503 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet fe523f0c364f447cb2a62b599d030cca with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:38.120833 3793 tablet_replica.cc:330] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:38.121044 3793 raft_consensus.cc:2229] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.121253 3793 raft_consensus.cc:2258] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.121697 3793 ts_tablet_manager.cc:1875] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.124032 3793 ts_tablet_manager.cc:1888] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:00:38.124111 3793 log.cc:1196] T fe523f0c364f447cb2a62b599d030cca P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/fe523f0c364f447cb2a62b599d030cca
I20240502 14:00:38.124653 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet fe523f0c364f447cb2a62b599d030cca (table rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]) successfully deleted
I20240502 14:00:38.169518 29863 consensus_queue.cc:237] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
W20240502 14:00:38.171183 29797 consensus_peers.cc:485] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c687dbe084f7469fa0167fc769b5de65. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.171221 29470 raft_consensus.cc:1261] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.171635 3830 consensus_queue.cc:1035] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.171710 29732 raft_consensus.cc:1261] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.172035 3830 consensus_queue.cc:1035] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.173715 2740 raft_consensus.cc:2932] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:38.174084 29732 raft_consensus.cc:2932] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:38.176615 29320 catalog_manager.cc:5484] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:38.176576 29470 raft_consensus.cc:2932] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:38.189232 29320 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:38.189409 29320 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:38.191645 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet ecb724afc4d545f8a709d34c8d5b72be
I20240502 14:00:38.192967 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb724afc4d545f8a709d34c8d5b72be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:38 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:38.193163 31841 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb724afc4d545f8a709d34c8d5b72be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:38 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:38.193205 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb724afc4d545f8a709d34c8d5b72be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:38 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:38.193341 3843 tablet_replica.cc:330] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:38.193506 3843 raft_consensus.cc:2229] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.193676 3844 tablet_replica.cc:330] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:38.193800 3845 tablet_replica.cc:330] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:38.193876 3844 raft_consensus.cc:2229] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.194065 3844 raft_consensus.cc:2258] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.193682 3843 raft_consensus.cc:2258] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.194666 3845 raft_consensus.cc:2229] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:38.194900 3844 ts_tablet_manager.cc:1875] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:38.195061 3845 raft_consensus.cc:2258] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.195221 3843 ts_tablet_manager.cc:1875] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:38.195735 3845 ts_tablet_manager.cc:1875] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:38.197764 3844 ts_tablet_manager.cc:1888] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:38.197901 3844 log.cc:1196] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/ecb724afc4d545f8a709d34c8d5b72be
I20240502 14:00:38.198244 3844 ts_tablet_manager.cc:1909] T ecb724afc4d545f8a709d34c8d5b72be P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:38.198573 3843 ts_tablet_manager.cc:1888] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:38.198575 3845 ts_tablet_manager.cc:1888] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:38.198649 3845 log.cc:1196] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/ecb724afc4d545f8a709d34c8d5b72be
I20240502 14:00:38.198756 3843 log.cc:1196] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/ecb724afc4d545f8a709d34c8d5b72be
I20240502 14:00:38.198952 3845 ts_tablet_manager.cc:1909] T ecb724afc4d545f8a709d34c8d5b72be P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:38.199088 3843 ts_tablet_manager.cc:1909] T ecb724afc4d545f8a709d34c8d5b72be P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:38.199087 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet ecb724afc4d545f8a709d34c8d5b72be (table rebalancer_dynamic_table_new_name [id=7b3b343cd3234696bb95fb0979489b76]) successfully deleted
I20240502 14:00:38.199744 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet ecb724afc4d545f8a709d34c8d5b72be (table rebalancer_dynamic_table_new_name [id=7b3b343cd3234696bb95fb0979489b76]) successfully deleted
I20240502 14:00:38.199995 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet ecb724afc4d545f8a709d34c8d5b72be (table rebalancer_dynamic_table_new_name [id=7b3b343cd3234696bb95fb0979489b76]) successfully deleted
I20240502 14:00:38.203583 29732 consensus_queue.cc:237] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.205044 29601 raft_consensus.cc:1261] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.205045 29863 raft_consensus.cc:1261] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.205446 3852 consensus_queue.cc:1035] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:38.205657 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d2ed5a4845b4bd78b19af6380c77982. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.205830 3599 consensus_queue.cc:1035] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.207885 3599 raft_consensus.cc:2932] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:38.208195 29601 raft_consensus.cc:2932] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:38.209995 29863 raft_consensus.cc:2932] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:38.212128 29325 catalog_manager.cc:5484] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:38.233422 29863 consensus_queue.cc:237] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.234869 29601 raft_consensus.cc:1261] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.234939 29732 raft_consensus.cc:1261] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:38.234987 29799 consensus_peers.cc:485] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b8885e2c89645d78b65b8a486ecd06b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.235297 2740 consensus_queue.cc:1035] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.235476 3671 consensus_queue.cc:1035] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.237715 3671 raft_consensus.cc:2932] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:38.238056 29601 raft_consensus.cc:2932] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:38.239535 29732 raft_consensus.cc:2932] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:38.241672 29327 catalog_manager.cc:5484] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:38.262966 29732 consensus_queue.cc:237] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:38.264497 29470 raft_consensus.cc:1261] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:38.264623 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1089cd67cbd24551836ccb702ff44d4a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.264658 29863 raft_consensus.cc:1261] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.264969 3584 consensus_queue.cc:1035] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.265421 3584 consensus_queue.cc:1035] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.267522 3599 raft_consensus.cc:2932] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:38.267959 29863 raft_consensus.cc:2932] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:38.269587 29470 raft_consensus.cc:2932] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:38.270831 29325 catalog_manager.cc:5484] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:38.290179 3575 consensus_queue.cc:1035] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.294148 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:38.295755 3904 consensus_queue.cc:1035] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.295887 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:38.301647 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 53696bdff96e45ce971747d938377e0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb]), partition=
I20240502 14:00:38.301894 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 53696bdff96e45ce971747d938377e0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb]), partition=
I20240502 14:00:38.301971 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53696bdff96e45ce971747d938377e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.302187 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53696bdff96e45ce971747d938377e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.304167 3907 tablet_bootstrap.cc:492] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:38.304651 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 53696bdff96e45ce971747d938377e0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb]), partition=
I20240502 14:00:38.304917 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53696bdff96e45ce971747d938377e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.305150 3907 tablet_bootstrap.cc:654] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.306829 3910 tablet_bootstrap.cc:492] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:38.307700 3907 tablet_bootstrap.cc:492] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:38.307875 3907 ts_tablet_manager.cc:1367] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:38.308017 3910 tablet_bootstrap.cc:654] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.308358 3907 raft_consensus.cc:348] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.308564 3907 raft_consensus.cc:374] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.308634 3907 raft_consensus.cc:729] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.308787 3907 consensus_queue.cc:260] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.309216 3907 ts_tablet_manager.cc:1398] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.312148 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 39db58bc129046bbbefea02b3eee4243 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.312454 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39db58bc129046bbbefea02b3eee4243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.313282 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 768df43846d94951a3a48f6f626bb473 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.313283 31841 tablet_service.cc:1467] Processing CreateTablet for tablet 39db58bc129046bbbefea02b3eee4243 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.313426 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1e00b442df441da78bbf2fcda5be73 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.313532 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768df43846d94951a3a48f6f626bb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.313547 31841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39db58bc129046bbbefea02b3eee4243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.313673 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1e00b442df441da78bbf2fcda5be73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.316169 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 768df43846d94951a3a48f6f626bb473 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.316407 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768df43846d94951a3a48f6f626bb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.316457 3913 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:38.316466 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 39db58bc129046bbbefea02b3eee4243 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.316711 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39db58bc129046bbbefea02b3eee4243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.316987 3907 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:38.317508 3913 tablet_bootstrap.cc:654] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.317934 3907 tablet_bootstrap.cc:654] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.318641 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1e00b442df441da78bbf2fcda5be73 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.318945 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1e00b442df441da78bbf2fcda5be73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.319141 3916 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:38.320360 3910 tablet_bootstrap.cc:492] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:38.320467 3910 ts_tablet_manager.cc:1367] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240502 14:00:38.320557 3916 tablet_bootstrap.cc:654] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.321002 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 768df43846d94951a3a48f6f626bb473 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.321235 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768df43846d94951a3a48f6f626bb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.321398 3910 raft_consensus.cc:348] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.321525 3910 raft_consensus.cc:374] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.321563 3910 raft_consensus.cc:729] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.321702 3910 consensus_queue.cc:260] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.322010 3910 ts_tablet_manager.cc:1398] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:38.322832 3907 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:38.322968 3907 ts_tablet_manager.cc:1367] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 14:00:38.323041 3913 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:38.323134 3913 ts_tablet_manager.cc:1367] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:38.323523 3907 raft_consensus.cc:348] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.323689 3907 raft_consensus.cc:374] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.326531 3913 raft_consensus.cc:348] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.326645 3913 raft_consensus.cc:374] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.326676 3913 raft_consensus.cc:729] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.326788 3913 consensus_queue.cc:260] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.327281 3913 ts_tablet_manager.cc:1398] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:38.327489 3913 tablet_bootstrap.cc:492] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:38.328361 3913 tablet_bootstrap.cc:654] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.330391 3913 tablet_bootstrap.cc:492] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:38.330480 3913 ts_tablet_manager.cc:1367] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:38.330976 3913 raft_consensus.cc:348] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.331223 3913 raft_consensus.cc:374] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.331312 3913 raft_consensus.cc:729] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.331488 3913 consensus_queue.cc:260] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.331840 3913 ts_tablet_manager.cc:1398] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.332068 3913 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:38.333009 3912 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:38.333030 3913 tablet_bootstrap.cc:654] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.333691 3908 tablet_bootstrap.cc:492] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:38.335139 3916 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:38.335353 3916 ts_tablet_manager.cc:1367] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:00:38.325366 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1e00b442df441da78bbf2fcda5be73 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.335917 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1e00b442df441da78bbf2fcda5be73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.336010 3916 raft_consensus.cc:348] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.336220 3916 raft_consensus.cc:374] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.336257 3916 raft_consensus.cc:729] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.336369 3916 consensus_queue.cc:260] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.336616 3916 ts_tablet_manager.cc:1398] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.336944 3912 tablet_bootstrap.cc:654] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.337723 3916 tablet_bootstrap.cc:492] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:38.338706 3916 tablet_bootstrap.cc:654] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.339937 3913 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:38.340021 3913 ts_tablet_manager.cc:1367] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:38.340430 3913 raft_consensus.cc:348] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.340523 3913 raft_consensus.cc:374] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.340550 3913 raft_consensus.cc:729] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.340675 3908 tablet_bootstrap.cc:654] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.340657 3913 consensus_queue.cc:260] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.340935 3913 ts_tablet_manager.cc:1398] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.323727 3907 raft_consensus.cc:729] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.344595 3907 consensus_queue.cc:260] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.347728 3907 ts_tablet_manager.cc:1398] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.025s user 0.000s sys 0.001s
I20240502 14:00:38.349057 3907 tablet_bootstrap.cc:492] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:38.349282 3912 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:38.349512 3912 ts_tablet_manager.cc:1367] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:38.351004 3912 raft_consensus.cc:348] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.351239 3912 raft_consensus.cc:374] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.351343 3912 raft_consensus.cc:729] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.351526 3912 consensus_queue.cc:260] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.351936 3907 tablet_bootstrap.cc:654] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.352697 3912 ts_tablet_manager.cc:1398] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:38.353536 3908 tablet_bootstrap.cc:492] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:38.354420 3908 ts_tablet_manager.cc:1367] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240502 14:00:38.355043 3908 raft_consensus.cc:348] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.355180 3908 raft_consensus.cc:374] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.355229 3908 raft_consensus.cc:729] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.355391 3908 consensus_queue.cc:260] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.356707 3908 ts_tablet_manager.cc:1398] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:38.359608 3916 tablet_bootstrap.cc:492] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:38.359714 3916 ts_tablet_manager.cc:1367] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20240502 14:00:38.360214 3916 raft_consensus.cc:348] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.360347 3916 raft_consensus.cc:374] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.360389 3916 raft_consensus.cc:729] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.360517 3916 consensus_queue.cc:260] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.360817 3916 ts_tablet_manager.cc:1398] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:38.360994 3916 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:38.362021 3916 tablet_bootstrap.cc:654] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.363965 3916 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:38.364130 3916 ts_tablet_manager.cc:1367] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:38.364991 3907 tablet_bootstrap.cc:492] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:38.364975 3916 raft_consensus.cc:348] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.365146 3907 ts_tablet_manager.cc:1367] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240502 14:00:38.365159 3916 raft_consensus.cc:374] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.365474 3916 raft_consensus.cc:729] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.365657 3916 consensus_queue.cc:260] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.365644 3907 raft_consensus.cc:348] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.366586 3907 raft_consensus.cc:374] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.366935 3907 raft_consensus.cc:729] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.366940 3916 ts_tablet_manager.cc:1398] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:38.367144 3907 consensus_queue.cc:260] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.368037 3907 ts_tablet_manager.cc:1398] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:38.374923 3830 raft_consensus.cc:482] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.375062 3830 raft_consensus.cc:504] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.375521 3830 leader_election.cc:288] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:38.376140 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53696bdff96e45ce971747d938377e0b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:38.376180 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53696bdff96e45ce971747d938377e0b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:38.376291 31990 raft_consensus.cc:2454] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:38.376292 29470 raft_consensus.cc:2454] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:38.376657 3685 raft_consensus.cc:482] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.376703 29799 leader_election.cc:302] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:38.376770 3685 raft_consensus.cc:504] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.376856 3830 raft_consensus.cc:2781] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.376901 3830 raft_consensus.cc:482] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.376933 3830 raft_consensus.cc:3037] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.377177 3685 leader_election.cc:288] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:38.377784 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db58bc129046bbbefea02b3eee4243" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:38.377902 32120 raft_consensus.cc:2454] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:38.378111 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db58bc129046bbbefea02b3eee4243" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:38.378223 31861 raft_consensus.cc:2454] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:38.378199 3830 raft_consensus.cc:504] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.378552 3830 leader_election.cc:288] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:38.378580 32188 leader_election.cc:302] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:38.378873 3685 raft_consensus.cc:2781] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.378933 3685 raft_consensus.cc:482] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.378970 3685 raft_consensus.cc:3037] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.378966 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53696bdff96e45ce971747d938377e0b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:38.379094 29470 raft_consensus.cc:3037] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.380040 29470 raft_consensus.cc:2454] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:38.380039 3685 raft_consensus.cc:504] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.380316 29799 leader_election.cc:302] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:38.380373 3685 leader_election.cc:288] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:38.380688 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db58bc129046bbbefea02b3eee4243" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:38.380808 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db58bc129046bbbefea02b3eee4243" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:38.380923 31861 raft_consensus.cc:3037] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.380822 32120 raft_consensus.cc:3037] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.381429 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53696bdff96e45ce971747d938377e0b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:38.381541 31990 raft_consensus.cc:3037] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.381951 31861 raft_consensus.cc:2454] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:38.382081 32120 raft_consensus.cc:2454] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:38.382398 32188 leader_election.cc:302] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:38.382634 31990 raft_consensus.cc:2454] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:38.382717 3685 raft_consensus.cc:2781] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.382794 3685 raft_consensus.cc:686] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:38.382956 3685 consensus_queue.cc:237] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.385929 3830 raft_consensus.cc:2781] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.386034 3830 raft_consensus.cc:686] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:38.385964 29327 catalog_manager.cc:5484] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:38.386169 3830 consensus_queue.cc:237] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:38.388394 29325 catalog_manager.cc:5484] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:38.490986 3904 raft_consensus.cc:482] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.491144 3904 raft_consensus.cc:504] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.491595 3904 leader_election.cc:288] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:38.492173 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768df43846d94951a3a48f6f626bb473" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:38.492188 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768df43846d94951a3a48f6f626bb473" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:38.492313 31861 raft_consensus.cc:2454] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:38.492326 32120 raft_consensus.cc:2454] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:38.492616 31925 leader_election.cc:302] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:38.492810 3904 raft_consensus.cc:2781] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.492875 3904 raft_consensus.cc:482] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.492925 3904 raft_consensus.cc:3037] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.493851 3904 raft_consensus.cc:504] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.494261 3904 leader_election.cc:288] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:38.494664 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768df43846d94951a3a48f6f626bb473" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:38.494676 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768df43846d94951a3a48f6f626bb473" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:38.494783 31861 raft_consensus.cc:3037] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.494786 32120 raft_consensus.cc:3037] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.495626 32120 raft_consensus.cc:2454] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:38.495811 31861 raft_consensus.cc:2454] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:38.495985 31925 leader_election.cc:302] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:38.496145 3904 raft_consensus.cc:2781] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.496222 3904 raft_consensus.cc:686] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:38.496347 3904 consensus_queue.cc:237] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.498904 29324 catalog_manager.cc:5484] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:38.501349 29324 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:38.505362 29470 raft_consensus.cc:1261] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:38.505604 31990 raft_consensus.cc:1261] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:38.505772 3830 consensus_queue.cc:1035] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.506026 2740 consensus_queue.cc:1035] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.508525 29324 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet 53696bdff96e45ce971747d938377e0b (table rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb]). Expected version 1 got 0
I20240502 14:00:38.509810 29325 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 53696bdff96e45ce971747d938377e0b (table rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb]). Expected version 1 got 0
I20240502 14:00:38.510769 3964 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:38.511169 3964 tablet.cc:1620] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:38.514570 3963 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:38.514039 3965 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:38.515260 3965 tablet.cc:1620] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:38.515305 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb] alter complete (version 1)
I20240502 14:00:38.515731 3963 tablet.cc:1620] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:38.516162 29325 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet 53696bdff96e45ce971747d938377e0b (table rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb]). Expected version 1 got 0
I20240502 14:00:38.557163 3966 ts_tablet_manager.cc:902] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:38.557762 3966 tablet_copy_client.cc:308] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:38.558234 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet c687dbe084f7469fa0167fc769b5de65 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:42159)
I20240502 14:00:38.558336 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet c687dbe084f7469fa0167fc769b5de65 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:42159: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-c687dbe084f7469fa0167fc769b5de65
I20240502 14:00:38.559338 29884 tablet_copy_source_session.cc:205] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.560320 3966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c687dbe084f7469fa0167fc769b5de65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.562175 3966 tablet_copy_client.cc:783] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.562314 3966 tablet_copy_client.cc:647] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.563604 3966 tablet_copy_client.cc:523] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.564889 3966 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:38.585672 3966 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.586045 3966 tablet_bootstrap.cc:492] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:38.586321 3966 ts_tablet_manager.cc:1367] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.006s
I20240502 14:00:38.586898 3966 raft_consensus.cc:348] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.587064 3966 raft_consensus.cc:729] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.587247 3966 consensus_queue.cc:260] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.587563 3966 ts_tablet_manager.cc:1398] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.588260 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-c687dbe084f7469fa0167fc769b5de65 received from {username='slave'} at 127.22.55.134:42159
I20240502 14:00:38.588374 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-c687dbe084f7469fa0167fc769b5de65 on tablet c687dbe084f7469fa0167fc769b5de65 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:38.618599 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:38.624547 3963 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:38.624943 3963 tablet.cc:1620] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:38.625022 3964 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:38.625476 3964 tablet.cc:1620] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:38.625458 3965 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:38.625759 3965 tablet.cc:1620] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:38.631033 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=a59ed63e46eb4c71ba4719c0cb47c8fb] alter complete (version 2)
I20240502 14:00:38.645187 3972 ts_tablet_manager.cc:902] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:38.645717 3972 tablet_copy_client.cc:308] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:38.646224 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 1089cd67cbd24551836ccb702ff44d4a from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:57583)
I20240502 14:00:38.646329 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 1089cd67cbd24551836ccb702ff44d4a from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:57583: session id = f0e01c4f581843e9ad4ced97f0072dc4-1089cd67cbd24551836ccb702ff44d4a
I20240502 14:00:38.647199 29753 tablet_copy_source_session.cc:205] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.648130 3972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1089cd67cbd24551836ccb702ff44d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.650235 3972 tablet_copy_client.cc:783] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.650393 3972 tablet_copy_client.cc:647] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.651537 3972 tablet_copy_client.cc:523] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.652737 3972 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:38.671948 3972 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.672271 3972 tablet_bootstrap.cc:492] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:38.672464 3972 ts_tablet_manager.cc:1367] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240502 14:00:38.672987 3972 raft_consensus.cc:348] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:38.673121 3972 raft_consensus.cc:729] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.673276 3972 consensus_queue.cc:260] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:38.673570 3972 ts_tablet_manager.cc:1398] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:38.674211 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-1089cd67cbd24551836ccb702ff44d4a received from {username='slave'} at 127.22.55.133:57583
I20240502 14:00:38.674299 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-1089cd67cbd24551836ccb702ff44d4a on tablet 1089cd67cbd24551836ccb702ff44d4a with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:38.708079 3904 raft_consensus.cc:482] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.708226 3904 raft_consensus.cc:504] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.708619 3904 leader_election.cc:288] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:38.709081 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1e00b442df441da78bbf2fcda5be73" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:38.709098 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1e00b442df441da78bbf2fcda5be73" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:38.709218 31861 raft_consensus.cc:2454] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:38.709232 32120 raft_consensus.cc:2454] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:38.709499 31925 leader_election.cc:302] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:38.709739 3904 raft_consensus.cc:2781] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.709791 3904 raft_consensus.cc:482] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.709848 3904 raft_consensus.cc:3037] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.710891 3904 raft_consensus.cc:504] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.711227 3904 leader_election.cc:288] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:38.711709 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1e00b442df441da78bbf2fcda5be73" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:38.711808 31861 raft_consensus.cc:3037] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.712591 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1e00b442df441da78bbf2fcda5be73" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:38.712698 32120 raft_consensus.cc:3037] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.712846 31861 raft_consensus.cc:2454] T 5e1e00b442df441da78bbf2fcda5be73 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:38.713219 31925 leader_election.cc:302] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:38.713376 3904 raft_consensus.cc:2781] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.713441 3904 raft_consensus.cc:686] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:38.713580 3904 consensus_queue.cc:237] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:38.713711 32120 raft_consensus.cc:2454] T 5e1e00b442df441da78bbf2fcda5be73 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:38.716063 29325 catalog_manager.cc:5484] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:38.735574 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:38.743814 29844 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:38.744936 31971 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:38.748610 3963 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:38.752756 3965 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:38.759516 3964 tablet.cc:1722] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:38.759940 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=a59ed63e46eb4c71ba4719c0cb47c8fb] alter complete (version 3)
I20240502 14:00:38.761760 32233 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:38.762732 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:38.766311 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.775720 32102 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:38.784890 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:38.793309 31839 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:38.815055 3997 ts_tablet_manager.cc:902] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:38.815732 3997 tablet_copy_client.cc:308] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:38.816223 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 3d2ed5a4845b4bd78b19af6380c77982 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:55687)
I20240502 14:00:38.816322 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 3d2ed5a4845b4bd78b19af6380c77982 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:55687: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-3d2ed5a4845b4bd78b19af6380c77982
I20240502 14:00:38.817291 29753 tablet_copy_source_session.cc:205] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.818182 3997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2ed5a4845b4bd78b19af6380c77982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.820375 3997 tablet_copy_client.cc:783] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.820526 3997 tablet_copy_client.cc:647] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.825033 3997 tablet_copy_client.cc:523] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.826570 4003 ts_tablet_manager.cc:902] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:38.827162 3997 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:38.827158 4003 tablet_copy_client.cc:308] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:38.827646 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 2b8885e2c89645d78b65b8a486ecd06b from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:34141)
I20240502 14:00:38.827762 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 2b8885e2c89645d78b65b8a486ecd06b from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:34141: session id = 8973f2527acb48f5ad36589652de73e1-2b8885e2c89645d78b65b8a486ecd06b
I20240502 14:00:38.828872 29884 tablet_copy_source_session.cc:205] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.829914 4003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8885e2c89645d78b65b8a486ecd06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.832108 4003 tablet_copy_client.cc:783] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.832270 4003 tablet_copy_client.cc:647] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.832875 3685 consensus_queue.cc:1035] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.836596 4003 tablet_copy_client.cc:523] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.836784 3685 consensus_queue.cc:1035] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.838088 4003 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:38.863248 3997 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.863601 3997 tablet_bootstrap.cc:492] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:38.863787 3997 ts_tablet_manager.cc:1367] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.000s
I20240502 14:00:38.864270 3997 raft_consensus.cc:348] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.864405 3997 raft_consensus.cc:729] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.864523 3997 consensus_queue.cc:260] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:38.864581 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:38.864732 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:38.864921 3997 ts_tablet_manager.cc:1398] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:38.865648 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-3d2ed5a4845b4bd78b19af6380c77982 received from {username='slave'} at 127.22.55.135:55687
I20240502 14:00:38.865722 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-3d2ed5a4845b4bd78b19af6380c77982 on tablet 3d2ed5a4845b4bd78b19af6380c77982 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:38.867678 4003 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.867964 4003 tablet_bootstrap.cc:492] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:38.868136 4003 ts_tablet_manager.cc:1367] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20240502 14:00:38.869189 4003 raft_consensus.cc:348] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.869319 4003 raft_consensus.cc:729] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.869446 4003 consensus_queue.cc:260] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:38.869773 4003 ts_tablet_manager.cc:1398] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:38.870312 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-2b8885e2c89645d78b65b8a486ecd06b received from {username='slave'} at 127.22.55.136:34141
I20240502 14:00:38.870391 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-2b8885e2c89645d78b65b8a486ecd06b on tablet 2b8885e2c89645d78b65b8a486ecd06b with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:38.870831 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 53696bdff96e45ce971747d938377e0b
I20240502 14:00:38.871692 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet 53696bdff96e45ce971747d938377e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:38 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:38.871692 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 53696bdff96e45ce971747d938377e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:38 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:38.872027 4028 tablet_replica.cc:330] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:38.872189 4028 raft_consensus.cc:2229] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.872316 4028 raft_consensus.cc:2258] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.872572 4027 tablet_replica.cc:330] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:38.873169 4028 ts_tablet_manager.cc:1875] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:38.874835 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet 53696bdff96e45ce971747d938377e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:38 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:38.875829 4028 ts_tablet_manager.cc:1888] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:38.876063 4027 raft_consensus.cc:2229] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:38.876319 4034 tablet_replica.cc:330] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:38.876458 4034 raft_consensus.cc:2229] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.876490 4027 raft_consensus.cc:2258] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.876590 4034 raft_consensus.cc:2258] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.876667 4028 log.cc:1196] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/53696bdff96e45ce971747d938377e0b
I20240502 14:00:38.877188 4028 ts_tablet_manager.cc:1909] T 53696bdff96e45ce971747d938377e0b P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:38.877199 4034 ts_tablet_manager.cc:1875] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:38.877868 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 53696bdff96e45ce971747d938377e0b (table rebalancer_dynamic_table_new_name [id=a59ed63e46eb4c71ba4719c0cb47c8fb]) successfully deleted
I20240502 14:00:38.878186 4027 ts_tablet_manager.cc:1875] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:38.880127 4034 ts_tablet_manager.cc:1888] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:38.880213 4034 log.cc:1196] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/53696bdff96e45ce971747d938377e0b
I20240502 14:00:38.880308 4027 ts_tablet_manager.cc:1888] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:38.880385 4027 log.cc:1196] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/53696bdff96e45ce971747d938377e0b
I20240502 14:00:38.880570 4034 ts_tablet_manager.cc:1909] T 53696bdff96e45ce971747d938377e0b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:38.880740 4027 ts_tablet_manager.cc:1909] T 53696bdff96e45ce971747d938377e0b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:38.887058 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 53696bdff96e45ce971747d938377e0b (table rebalancer_dynamic_table_new_name [id=a59ed63e46eb4c71ba4719c0cb47c8fb]) successfully deleted
I20240502 14:00:38.887063 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 53696bdff96e45ce971747d938377e0b (table rebalancer_dynamic_table_new_name [id=a59ed63e46eb4c71ba4719c0cb47c8fb]) successfully deleted
I20240502 14:00:38.960803 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.961627 3904 consensus_queue.cc:1035] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.967388 32233 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:38.972136 32102 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:38.974964 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:38.975040 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:38.975591 29324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:38.981731 3575 consensus_queue.cc:1035] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.983691 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb9cd5aa50460bbed5d3984e352bba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a]), partition=
I20240502 14:00:38.983691 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb9cd5aa50460bbed5d3984e352bba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a]), partition=
I20240502 14:00:38.983994 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb9cd5aa50460bbed5d3984e352bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.983994 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb9cd5aa50460bbed5d3984e352bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.987035 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb9cd5aa50460bbed5d3984e352bba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a]), partition=
I20240502 14:00:38.987365 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb9cd5aa50460bbed5d3984e352bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.000355 4073 tablet_bootstrap.cc:492] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:39.002445 4073 tablet_bootstrap.cc:654] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.003940 31971 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:39.004227 31839 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:39.008512 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:39.009199 4076 tablet_bootstrap.cc:492] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:39.011401 4076 tablet_bootstrap.cc:654] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.013423 4074 tablet_bootstrap.cc:492] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:39.030483 4074 tablet_bootstrap.cc:654] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.036789 4074 tablet_bootstrap.cc:492] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:39.036914 4074 ts_tablet_manager.cc:1367] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 14:00:39.037451 4074 raft_consensus.cc:348] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.037578 4074 raft_consensus.cc:374] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.037612 4074 raft_consensus.cc:729] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.037786 4074 consensus_queue.cc:260] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.038107 4074 ts_tablet_manager.cc:1398] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:39.041313 4073 tablet_bootstrap.cc:492] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:39.041414 4073 ts_tablet_manager.cc:1367] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240502 14:00:39.041954 4073 raft_consensus.cc:348] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.042096 4073 raft_consensus.cc:374] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.042146 4073 raft_consensus.cc:729] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.042274 4073 consensus_queue.cc:260] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.042539 4073 ts_tablet_manager.cc:1398] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.046922 4076 tablet_bootstrap.cc:492] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:39.047029 4076 ts_tablet_manager.cc:1367] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240502 14:00:39.047492 4076 raft_consensus.cc:348] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.047606 4076 raft_consensus.cc:374] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.047641 4076 raft_consensus.cc:729] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.047801 4076 consensus_queue.cc:260] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.048106 4076 ts_tablet_manager.cc:1398] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:39.124883 31861 raft_consensus.cc:1203] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:39.141324 31992 raft_consensus.cc:1203] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:39.194983 3996 raft_consensus.cc:482] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:39.195132 3996 raft_consensus.cc:504] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.195483 3996 leader_election.cc:288] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:39.195819 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb9cd5aa50460bbed5d3984e352bba" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:39.195966 32255 raft_consensus.cc:2454] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:39.196235 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb9cd5aa50460bbed5d3984e352bba" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:39.196282 32058 leader_election.cc:302] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:39.196374 29601 raft_consensus.cc:2454] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:39.196463 3996 raft_consensus.cc:2781] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:39.196511 3996 raft_consensus.cc:482] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:39.196537 3996 raft_consensus.cc:3037] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.197398 3996 raft_consensus.cc:504] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.197711 3996 leader_election.cc:288] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:39.198004 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb9cd5aa50460bbed5d3984e352bba" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:39.198032 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb9cd5aa50460bbed5d3984e352bba" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:39.198127 32255 raft_consensus.cc:3037] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.198168 29601 raft_consensus.cc:3037] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.199198 29601 raft_consensus.cc:2454] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:39.199199 32255 raft_consensus.cc:2454] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:39.199479 32058 leader_election.cc:302] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:39.199669 3996 raft_consensus.cc:2781] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:39.199764 3996 raft_consensus.cc:686] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:39.200090 3996 consensus_queue.cc:237] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.202121 29322 catalog_manager.cc:5484] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:39.221935 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.233116 32233 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:39.238521 31971 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:39.239238 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:39.252197 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.255357 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.264182 31839 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:39.264676 32102 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.266516 3575 consensus_queue.cc:1035] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.273670 3575 consensus_queue.cc:1035] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.277392 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:39.294147 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 824fb738a4b84e13b5fc283a26c92355 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.294507 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824fb738a4b84e13b5fc283a26c92355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.296762 4136 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:39.297012 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 75347e9c867547efaf78355aec6dfb40 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.297215 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75347e9c867547efaf78355aec6dfb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.297664 4136 tablet_bootstrap.cc:654] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.301131 4136 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:39.301213 4136 ts_tablet_manager.cc:1367] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:39.302668 4136 raft_consensus.cc:348] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.302877 4136 raft_consensus.cc:374] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.302914 4136 raft_consensus.cc:729] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.303027 4136 consensus_queue.cc:260] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.303750 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 7c83270b40d541939325d058bcf807e4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.304026 4136 ts_tablet_manager.cc:1398] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:39.304052 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c83270b40d541939325d058bcf807e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.304232 4136 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:39.305290 4136 tablet_bootstrap.cc:654] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.308028 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 824fb738a4b84e13b5fc283a26c92355 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.308324 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824fb738a4b84e13b5fc283a26c92355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.309556 4136 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:39.309640 4136 ts_tablet_manager.cc:1367] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:39.310446 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 824fb738a4b84e13b5fc283a26c92355 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.310698 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824fb738a4b84e13b5fc283a26c92355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.310689 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 7c83270b40d541939325d058bcf807e4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.310935 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c83270b40d541939325d058bcf807e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.311045 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 75347e9c867547efaf78355aec6dfb40 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.311316 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75347e9c867547efaf78355aec6dfb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.313060 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 75347e9c867547efaf78355aec6dfb40 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.313336 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75347e9c867547efaf78355aec6dfb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.313277 4136 raft_consensus.cc:348] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.313408 4136 raft_consensus.cc:374] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.313441 4136 raft_consensus.cc:729] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.313503 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 7c83270b40d541939325d058bcf807e4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.313566 4136 consensus_queue.cc:260] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.313741 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c83270b40d541939325d058bcf807e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.313956 4136 ts_tablet_manager.cc:1398] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:00:39.315888 4074 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:39.316395 4076 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:39.316731 4141 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:39.316890 4074 tablet_bootstrap.cc:654] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.317517 4076 tablet_bootstrap.cc:654] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.319113 4141 tablet_bootstrap.cc:654] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.319132 4139 tablet_bootstrap.cc:492] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:39.319994 4139 tablet_bootstrap.cc:654] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.320463 4141 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:39.320544 4141 ts_tablet_manager.cc:1367] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:39.320989 4141 raft_consensus.cc:348] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.321094 4141 raft_consensus.cc:374] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.321122 4141 raft_consensus.cc:729] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.321252 4141 consensus_queue.cc:260] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.321506 4141 ts_tablet_manager.cc:1398] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.324445 32120 raft_consensus.cc:1203] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:39.327895 4076 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:39.328025 4076 ts_tablet_manager.cc:1367] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:39.328575 4076 raft_consensus.cc:348] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.328697 4076 raft_consensus.cc:374] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.328733 4076 raft_consensus.cc:729] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.328861 4076 consensus_queue.cc:260] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.328914 4139 tablet_bootstrap.cc:492] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:39.329000 4139 ts_tablet_manager.cc:1367] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 14:00:39.329469 4139 raft_consensus.cc:348] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.329558 4076 ts_tablet_manager.cc:1398] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.329665 4139 raft_consensus.cc:374] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.329756 4139 raft_consensus.cc:729] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.329895 4140 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:39.330008 4139 consensus_queue.cc:260] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.330209 4076 tablet_bootstrap.cc:492] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:39.330441 4139 ts_tablet_manager.cc:1398] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.331315 4076 tablet_bootstrap.cc:654] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.331976 4140 tablet_bootstrap.cc:654] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.333824 4074 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:39.333952 4074 ts_tablet_manager.cc:1367] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:00:39.334434 4074 raft_consensus.cc:348] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.334544 4074 raft_consensus.cc:374] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.334578 4074 raft_consensus.cc:729] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.334700 4074 consensus_queue.cc:260] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.334914 32255 raft_consensus.cc:1203] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:39.335009 4074 ts_tablet_manager.cc:1398] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.335191 4074 tablet_bootstrap.cc:492] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:39.336150 4074 tablet_bootstrap.cc:654] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.340773 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:39.341877 4074 tablet_bootstrap.cc:492] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:39.341984 4074 ts_tablet_manager.cc:1367] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:39.342067 4140 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:39.342162 4140 ts_tablet_manager.cc:1367] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:39.342535 4074 raft_consensus.cc:348] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.342610 4140 raft_consensus.cc:348] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.342671 4074 raft_consensus.cc:374] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.342717 4074 raft_consensus.cc:729] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.342741 4140 raft_consensus.cc:374] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.342775 4140 raft_consensus.cc:729] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.342891 4074 consensus_queue.cc:260] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.342908 4140 consensus_queue.cc:260] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.343201 4074 ts_tablet_manager.cc:1398] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.343732 4140 ts_tablet_manager.cc:1398] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:39.345248 29601 raft_consensus.cc:1261] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:39.345391 32255 raft_consensus.cc:1261] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:39.345752 3329 consensus_queue.cc:1035] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.346763 3329 consensus_queue.cc:1035] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.350802 29326 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 08fb9cd5aa50460bbed5d3984e352bba (table rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a]). Expected version 1 got 0
I20240502 14:00:39.351191 29322 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 08fb9cd5aa50460bbed5d3984e352bba (table rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a]). Expected version 1 got 0
I20240502 14:00:39.353058 4153 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:39.353587 4153 tablet.cc:1620] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:39.353601 29322 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 08fb9cd5aa50460bbed5d3984e352bba (table rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a]). Expected version 1 got 0
I20240502 14:00:39.355638 4152 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:39.355974 4152 tablet.cc:1620] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:39.358973 3575 raft_consensus.cc:482] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:39.359094 3575 raft_consensus.cc:504] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.359472 3575 leader_election.cc:288] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:39.359923 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824fb738a4b84e13b5fc283a26c92355" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:39.360072 32255 raft_consensus.cc:2454] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:39.360376 4076 tablet_bootstrap.cc:492] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:39.360339 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824fb738a4b84e13b5fc283a26c92355" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:39.360471 4076 ts_tablet_manager.cc:1367] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 14:00:39.360486 29732 raft_consensus.cc:2454] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:39.360834 31926 leader_election.cc:302] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:39.360954 4076 raft_consensus.cc:348] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.361047 4076 raft_consensus.cc:374] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.361068 4076 raft_consensus.cc:729] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.361184 4076 consensus_queue.cc:260] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.361480 4076 ts_tablet_manager.cc:1398] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.361797 3575 raft_consensus.cc:2781] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:39.361865 3575 raft_consensus.cc:482] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:39.361897 3575 raft_consensus.cc:3037] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.362747 3575 raft_consensus.cc:504] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.363134 3575 leader_election.cc:288] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:39.363742 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824fb738a4b84e13b5fc283a26c92355" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:39.363867 32255 raft_consensus.cc:3037] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.363961 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a] alter complete (version 1)
I20240502 14:00:39.364367 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824fb738a4b84e13b5fc283a26c92355" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:39.364470 29732 raft_consensus.cc:3037] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.364941 32255 raft_consensus.cc:2454] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:39.365394 29732 raft_consensus.cc:2454] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:39.365406 31927 leader_election.cc:302] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:39.365710 3575 raft_consensus.cc:2781] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:39.365811 3575 raft_consensus.cc:686] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:39.365952 3575 consensus_queue.cc:237] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:39.367964 29326 catalog_manager.cc:5484] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:39.370434 29322 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 08fb9cd5aa50460bbed5d3984e352bba (table rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a]). Expected version 1 got 0
I20240502 14:00:39.372332 4158 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:39.372696 4158 tablet.cc:1620] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:39.391075 29862 consensus_queue.cc:237] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:39.392280 31992 raft_consensus.cc:1261] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.392411 29732 raft_consensus.cc:1261] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.392691 2740 consensus_queue.cc:1035] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.392843 3671 consensus_queue.cc:1035] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.394785 4174 raft_consensus.cc:2932] T c687dbe084f7469fa0167fc769b5de65 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:39.395207 29732 raft_consensus.cc:2932] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:39.397779 31992 raft_consensus.cc:2932] T c687dbe084f7469fa0167fc769b5de65 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:39.398217 29325 catalog_manager.cc:5484] T c687dbe084f7469fa0167fc769b5de65 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:39.401993 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet c687dbe084f7469fa0167fc769b5de65 with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:39.402360 4175 tablet_replica.cc:330] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:39.402472 3685 raft_consensus.cc:482] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:39.402530 4175 raft_consensus.cc:2229] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.402603 3685 raft_consensus.cc:504] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.402696 4175 raft_consensus.cc:2258] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.403060 3685 leader_election.cc:288] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:39.403607 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c83270b40d541939325d058bcf807e4" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:39.403648 4175 ts_tablet_manager.cc:1875] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.403755 31861 raft_consensus.cc:2454] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:39.404842 32188 leader_election.cc:302] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:39.405016 3685 raft_consensus.cc:2781] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:39.405078 3685 raft_consensus.cc:482] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:39.405115 3685 raft_consensus.cc:3037] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.405143 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c83270b40d541939325d058bcf807e4" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:39.405278 32120 raft_consensus.cc:2454] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:39.405902 4175 ts_tablet_manager.cc:1888] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:39.405957 4175 log.cc:1196] T c687dbe084f7469fa0167fc769b5de65 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/c687dbe084f7469fa0167fc769b5de65
I20240502 14:00:39.406150 3685 raft_consensus.cc:504] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.406487 3685 leader_election.cc:288] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:39.406493 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet c687dbe084f7469fa0167fc769b5de65 (table rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]) successfully deleted
I20240502 14:00:39.406967 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c83270b40d541939325d058bcf807e4" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:39.407086 32120 raft_consensus.cc:3037] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.407166 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c83270b40d541939325d058bcf807e4" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:39.407274 31861 raft_consensus.cc:3037] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.408077 32120 raft_consensus.cc:2454] T 7c83270b40d541939325d058bcf807e4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:39.408217 31861 raft_consensus.cc:2454] T 7c83270b40d541939325d058bcf807e4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:39.408509 32188 leader_election.cc:302] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:39.408679 3685 raft_consensus.cc:2781] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:39.408756 3685 raft_consensus.cc:686] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:39.408886 3685 consensus_queue.cc:237] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.411542 29321 catalog_manager.cc:5484] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:39.440433 3329 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:39.440560 3329 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.440927 3329 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:39.441323 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75347e9c867547efaf78355aec6dfb40" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:39.441345 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75347e9c867547efaf78355aec6dfb40" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:39.441483 31992 raft_consensus.cc:2454] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:39.441507 29862 raft_consensus.cc:2454] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:39.441803 32056 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:39.441974 3329 raft_consensus.cc:2781] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:39.442040 3329 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:39.442080 3329 raft_consensus.cc:3037] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.443219 3329 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.443583 3329 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:39.444152 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75347e9c867547efaf78355aec6dfb40" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:39.444295 29862 raft_consensus.cc:3037] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.444300 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75347e9c867547efaf78355aec6dfb40" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:39.444416 31992 raft_consensus.cc:3037] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.445362 31992 raft_consensus.cc:2454] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:39.445361 29862 raft_consensus.cc:2454] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:39.445686 32057 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:39.445914 3329 raft_consensus.cc:2781] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:39.446002 3329 raft_consensus.cc:686] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:39.446188 3329 consensus_queue.cc:237] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.448422 29321 catalog_manager.cc:5484] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:39.472487 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:39.479837 4152 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:39.480187 4152 tablet.cc:1620] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:39.480300 4158 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:39.480573 4158 tablet.cc:1620] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:39.484470 4153 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:39.484773 4153 tablet.cc:1620] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:39.493410 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=8985c2f5bcef4cd7ad1142026cf0b89a] alter complete (version 2)
I20240502 14:00:39.515892 31972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.519052 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.524971 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.531452 31839 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:39.538321 32103 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:39.539453 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.540175 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:39.547421 32235 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.595577 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:39.601791 4152 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:39.602142 4153 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:39.605080 4158 tablet.cc:1722] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:39.608127 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=8985c2f5bcef4cd7ad1142026cf0b89a] alter complete (version 3)
I20240502 14:00:39.623500 29601 consensus_queue.cc:237] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:39.625092 29471 raft_consensus.cc:1261] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.625092 31860 raft_consensus.cc:1261] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.625561 3935 consensus_queue.cc:1035] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.626029 3935 consensus_queue.cc:1035] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.628160 2862 raft_consensus.cc:2932] T 6eae206e12b44b809d8b2fcc21611a28 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:39.629365 29471 raft_consensus.cc:2932] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:39.630053 31860 raft_consensus.cc:2932] T 6eae206e12b44b809d8b2fcc21611a28 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:39.632484 29329 catalog_manager.cc:5484] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:39.635469 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eae206e12b44b809d8b2fcc21611a28 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:39.635828 4250 tablet_replica.cc:330] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:39.635995 4250 raft_consensus.cc:2229] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.636269 4250 raft_consensus.cc:2258] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.636833 4250 ts_tablet_manager.cc:1875] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.639122 4250 ts_tablet_manager.cc:1888] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:39.639209 4250 log.cc:1196] T 6eae206e12b44b809d8b2fcc21611a28 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/6eae206e12b44b809d8b2fcc21611a28
I20240502 14:00:39.639760 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 6eae206e12b44b809d8b2fcc21611a28 (table rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]) successfully deleted
I20240502 14:00:39.712965 29328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:39.713122 29328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:39.715826 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 08fb9cd5aa50460bbed5d3984e352bba
I20240502 14:00:39.716553 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fb9cd5aa50460bbed5d3984e352bba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:39 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:39.716699 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fb9cd5aa50460bbed5d3984e352bba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:39 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:39.717123 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fb9cd5aa50460bbed5d3984e352bba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:39 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:39.722304 4295 tablet_replica.cc:330] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:39.722018 4296 tablet_replica.cc:330] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:39.722615 4296 raft_consensus.cc:2229] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.722823 4295 raft_consensus.cc:2229] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:39.723225 4296 raft_consensus.cc:2258] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.723522 4295 raft_consensus.cc:2258] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.724381 4296 ts_tablet_manager.cc:1875] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:39.724828 4295 ts_tablet_manager.cc:1875] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:39.729533 4294 tablet_replica.cc:330] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:39.729775 4294 raft_consensus.cc:2229] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.729959 4294 raft_consensus.cc:2258] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.730543 4294 ts_tablet_manager.cc:1875] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:39.732030 4295 ts_tablet_manager.cc:1888] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:39.732153 4296 ts_tablet_manager.cc:1888] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:39.732288 4295 log.cc:1196] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/08fb9cd5aa50460bbed5d3984e352bba
I20240502 14:00:39.732403 4296 log.cc:1196] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/08fb9cd5aa50460bbed5d3984e352bba
I20240502 14:00:39.732728 4295 ts_tablet_manager.cc:1909] T 08fb9cd5aa50460bbed5d3984e352bba P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:39.732981 4294 ts_tablet_manager.cc:1888] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:39.733098 4294 log.cc:1196] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/08fb9cd5aa50460bbed5d3984e352bba
I20240502 14:00:39.733408 4294 ts_tablet_manager.cc:1909] T 08fb9cd5aa50460bbed5d3984e352bba P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:39.733559 4296 ts_tablet_manager.cc:1909] T 08fb9cd5aa50460bbed5d3984e352bba P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:39.734524 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 08fb9cd5aa50460bbed5d3984e352bba (table rebalancer_dynamic_table_new_name [id=8985c2f5bcef4cd7ad1142026cf0b89a]) successfully deleted
I20240502 14:00:39.735065 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 08fb9cd5aa50460bbed5d3984e352bba (table rebalancer_dynamic_table_new_name [id=8985c2f5bcef4cd7ad1142026cf0b89a]) successfully deleted
I20240502 14:00:39.735723 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 08fb9cd5aa50460bbed5d3984e352bba (table rebalancer_dynamic_table_new_name [id=8985c2f5bcef4cd7ad1142026cf0b89a]) successfully deleted
I20240502 14:00:39.747567 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.750718 32235 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.750990 31972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.751391 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.751690 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.763512 31839 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:39.769464 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:39.789184 32103 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:39.817989 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:39.823544 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 58cbe304ca0c4052a959b5840be3394b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b]), partition=
I20240502 14:00:39.823642 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 58cbe304ca0c4052a959b5840be3394b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b]), partition=
I20240502 14:00:39.823851 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58cbe304ca0c4052a959b5840be3394b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.823889 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58cbe304ca0c4052a959b5840be3394b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.825809 4074 tablet_bootstrap.cc:492] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:39.826025 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 58cbe304ca0c4052a959b5840be3394b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b]), partition=
I20240502 14:00:39.826263 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58cbe304ca0c4052a959b5840be3394b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.826737 4074 tablet_bootstrap.cc:654] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.828917 4310 tablet_bootstrap.cc:492] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:39.829350 4074 tablet_bootstrap.cc:492] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:39.829437 4074 ts_tablet_manager.cc:1367] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:39.829908 4074 raft_consensus.cc:348] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.829993 4308 tablet_bootstrap.cc:492] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:39.830016 4074 raft_consensus.cc:374] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.830049 4074 raft_consensus.cc:729] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.830164 4074 consensus_queue.cc:260] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.830484 4074 ts_tablet_manager.cc:1398] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:39.830682 4310 tablet_bootstrap.cc:654] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.830897 4308 tablet_bootstrap.cc:654] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.832414 4310 tablet_bootstrap.cc:492] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:39.832512 4310 ts_tablet_manager.cc:1367] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:39.832975 4310 raft_consensus.cc:348] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.833092 4310 raft_consensus.cc:374] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.833122 4310 raft_consensus.cc:729] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.833242 4310 consensus_queue.cc:260] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.833609 4308 tablet_bootstrap.cc:492] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:39.833695 4308 ts_tablet_manager.cc:1367] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:39.834118 4308 raft_consensus.cc:348] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.834223 4308 raft_consensus.cc:374] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.834267 4308 raft_consensus.cc:729] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.834378 4308 consensus_queue.cc:260] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:39.834929 4310 ts_tablet_manager.cc:1398] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:39.835274 4308 ts_tablet_manager.cc:1398] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:39.858639 29862 consensus_queue.cc:237] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:39.859259 3685 consensus_queue.cc:1035] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.860024 29732 raft_consensus.cc:1261] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.860188 32255 raft_consensus.cc:1261] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.860885 4174 consensus_queue.cc:1035] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.861157 4174 consensus_queue.cc:1035] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.863097 3685 consensus_queue.cc:1035] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.864327 3671 raft_consensus.cc:2932] T 2b8885e2c89645d78b65b8a486ecd06b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:39.864668 29732 raft_consensus.cc:2932] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:39.865145 32255 raft_consensus.cc:2932] T 2b8885e2c89645d78b65b8a486ecd06b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:39.867691 29328 catalog_manager.cc:5484] T 2b8885e2c89645d78b65b8a486ecd06b P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:39.870545 3904 consensus_queue.cc:1035] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.871974 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8885e2c89645d78b65b8a486ecd06b with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:39.872247 4296 tablet_replica.cc:330] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:39.872400 4296 raft_consensus.cc:2229] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.872548 4296 raft_consensus.cc:2258] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.873205 4296 ts_tablet_manager.cc:1875] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.874547 3904 consensus_queue.cc:1035] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.877945 4296 ts_tablet_manager.cc:1888] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:39.878111 4296 log.cc:1196] T 2b8885e2c89645d78b65b8a486ecd06b P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b8885e2c89645d78b65b8a486ecd06b
I20240502 14:00:39.878878 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 2b8885e2c89645d78b65b8a486ecd06b (table rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]) successfully deleted
I20240502 14:00:39.956171 3996 consensus_queue.cc:1035] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.958010 29325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:39.967284 3329 consensus_queue.cc:1035] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.972390 32235 tablet_service.cc:1467] Processing CreateTablet for tablet ea104c9432a7416f87760e7f6458d6cb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.972718 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea104c9432a7416f87760e7f6458d6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.972891 31972 tablet_service.cc:1467] Processing CreateTablet for tablet ea104c9432a7416f87760e7f6458d6cb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.973169 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea104c9432a7416f87760e7f6458d6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.973332 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 541806c11f074316a5342050427f4849 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.973565 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541806c11f074316a5342050427f4849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.974717 4308 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:39.975070 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 541806c11f074316a5342050427f4849 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.975319 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541806c11f074316a5342050427f4849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.975625 32103 tablet_service.cc:1467] Processing CreateTablet for tablet eab1144457d34c25897c5b7a691409fe (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.975732 4308 tablet_bootstrap.cc:654] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.975849 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab1144457d34c25897c5b7a691409fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.975930 31840 tablet_service.cc:1467] Processing CreateTablet for tablet eab1144457d34c25897c5b7a691409fe (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.976188 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab1144457d34c25897c5b7a691409fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.976691 31839 tablet_service.cc:1467] Processing CreateTablet for tablet ea104c9432a7416f87760e7f6458d6cb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.976933 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea104c9432a7416f87760e7f6458d6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.978433 4074 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:39.978627 4383 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:39.979034 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 541806c11f074316a5342050427f4849 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.979319 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541806c11f074316a5342050427f4849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.979405 32233 tablet_service.cc:1467] Processing CreateTablet for tablet eab1144457d34c25897c5b7a691409fe (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.979544 4074 tablet_bootstrap.cc:654] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.979640 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab1144457d34c25897c5b7a691409fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.981578 4387 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:39.983201 4387 tablet_bootstrap.cc:654] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.984114 4383 tablet_bootstrap.cc:654] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.984424 4385 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:39.984674 4308 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:39.984795 4308 ts_tablet_manager.cc:1367] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:39.985127 4074 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:39.985213 4074 ts_tablet_manager.cc:1367] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:39.985301 4308 raft_consensus.cc:348] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:39.985416 4308 raft_consensus.cc:374] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.985450 4308 raft_consensus.cc:729] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.985569 4308 consensus_queue.cc:260] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:39.985639 4074 raft_consensus.cc:348] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:39.985751 4074 raft_consensus.cc:374] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.985785 4074 raft_consensus.cc:729] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.985843 4308 ts_tablet_manager.cc:1398] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.985889 4074 consensus_queue.cc:260] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:39.986011 4308 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:39.986217 4385 tablet_bootstrap.cc:654] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.986980 4074 ts_tablet_manager.cc:1398] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:39.987210 4074 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:39.987727 4387 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:39.987826 4387 ts_tablet_manager.cc:1367] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:39.988080 4308 tablet_bootstrap.cc:654] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.988093 4074 tablet_bootstrap.cc:654] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.988303 4387 raft_consensus.cc:348] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:39.988420 4387 raft_consensus.cc:374] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.988461 4387 raft_consensus.cc:729] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.988582 4387 consensus_queue.cc:260] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:39.988883 4387 ts_tablet_manager.cc:1398] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.989137 4383 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:39.989261 4383 ts_tablet_manager.cc:1367] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240502 14:00:39.990471 4385 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:39.990921 4308 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:39.991005 4308 ts_tablet_manager.cc:1367] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:39.991466 4308 raft_consensus.cc:348] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:39.991196 4385 ts_tablet_manager.cc:1367] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:39.991570 4308 raft_consensus.cc:374] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.991602 4308 raft_consensus.cc:729] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.991719 4308 consensus_queue.cc:260] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:39.991999 4308 ts_tablet_manager.cc:1398] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.992264 4383 raft_consensus.cc:348] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:39.992363 4383 raft_consensus.cc:374] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.992396 4383 raft_consensus.cc:729] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.993126 4074 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:39.993202 4074 ts_tablet_manager.cc:1367] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 14:00:39.993139 4385 raft_consensus.cc:348] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:39.993261 4385 raft_consensus.cc:374] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.993297 4385 raft_consensus.cc:729] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.993459 4385 consensus_queue.cc:260] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:39.993729 4385 ts_tablet_manager.cc:1398] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:39.993963 4385 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:39.994988 4385 tablet_bootstrap.cc:654] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.994968 4383 consensus_queue.cc:260] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:39.995379 4383 ts_tablet_manager.cc:1398] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 14:00:39.995584 4383 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:39.996409 4385 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:39.996490 4385 ts_tablet_manager.cc:1367] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:39.994954 4074 raft_consensus.cc:348] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:39.996734 4383 tablet_bootstrap.cc:654] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.996760 4074 raft_consensus.cc:374] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.996814 4074 raft_consensus.cc:729] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.996960 4074 consensus_queue.cc:260] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:39.996959 4385 raft_consensus.cc:348] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:39.997076 4385 raft_consensus.cc:374] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.997118 4385 raft_consensus.cc:729] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.997247 4385 consensus_queue.cc:260] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:39.997567 4385 ts_tablet_manager.cc:1398] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:39.998103 4383 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:39.998212 4383 ts_tablet_manager.cc:1367] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:39.998649 4074 ts_tablet_manager.cc:1398] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:39.998673 4383 raft_consensus.cc:348] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:39.998783 4383 raft_consensus.cc:374] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.998824 4383 raft_consensus.cc:729] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.998999 4383 consensus_queue.cc:260] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:39.999406 4383 ts_tablet_manager.cc:1398] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:40.006206 4174 raft_consensus.cc:482] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:40.006320 4174 raft_consensus.cc:504] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:40.006695 4174 leader_election.cc:288] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:40.007134 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cbe304ca0c4052a959b5840be3394b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:40.007198 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cbe304ca0c4052a959b5840be3394b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:40.007277 32122 raft_consensus.cc:2454] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:40.007356 31990 raft_consensus.cc:2454] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:40.007601 29797 leader_election.cc:302] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.007920 4174 raft_consensus.cc:2781] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:40.007983 4174 raft_consensus.cc:482] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:40.008028 4174 raft_consensus.cc:3037] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.008975 4174 raft_consensus.cc:504] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:40.009295 4174 leader_election.cc:288] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:40.009688 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cbe304ca0c4052a959b5840be3394b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:40.009685 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cbe304ca0c4052a959b5840be3394b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:40.009805 32122 raft_consensus.cc:3037] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.009852 31990 raft_consensus.cc:3037] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.010674 31990 raft_consensus.cc:2454] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:40.010681 32122 raft_consensus.cc:2454] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:40.010968 29797 leader_election.cc:302] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:40.011137 4174 raft_consensus.cc:2781] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:40.011220 4174 raft_consensus.cc:686] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:40.011387 4174 consensus_queue.cc:237] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:40.013458 29329 catalog_manager.cc:5484] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:40.035472 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.039031 31972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.042330 32235 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.045405 29844 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:40.045456 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.047189 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:40.051169 31839 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:40.066566 32103 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.071152 4019 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:40.071266 4019 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.071663 4019 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:40.072207 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:40.072346 32122 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:40.072726 31794 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.072918 4019 raft_consensus.cc:2781] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:40.072968 4019 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:40.073001 4019 raft_consensus.cc:3037] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.074260 4019 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.074613 4019 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:40.074944 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:40.075058 32122 raft_consensus.cc:3037] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.075199 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:40.075318 32254 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:40.076052 32122 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:40.076292 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:40.076418 32255 raft_consensus.cc:3037] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.077077 32255 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:40.077332 31796 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:40.077522 4019 raft_consensus.cc:2781] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:40.077639 4019 raft_consensus.cc:686] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:40.078351 4019 consensus_queue.cc:237] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.080852 29322 catalog_manager.cc:5484] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:40.111873 4407 raft_consensus.cc:482] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:40.112001 4407 raft_consensus.cc:504] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.112613 4407 leader_election.cc:288] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:40.112782 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:40.112936 32255 raft_consensus.cc:2454] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:40.112964 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:40.113088 31990 raft_consensus.cc:2454] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:40.113286 31796 leader_election.cc:302] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:40.113504 4407 raft_consensus.cc:2781] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:40.113554 4407 raft_consensus.cc:482] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:40.113587 4407 raft_consensus.cc:3037] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.114774 4407 raft_consensus.cc:504] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.115139 4407 leader_election.cc:288] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:40.115478 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:40.115603 32255 raft_consensus.cc:3037] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.115598 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:40.115731 31990 raft_consensus.cc:3037] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.116678 32255 raft_consensus.cc:2454] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:40.116678 31990 raft_consensus.cc:2454] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:40.116990 31796 leader_election.cc:302] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:40.117261 4407 raft_consensus.cc:2781] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:40.117347 4407 raft_consensus.cc:686] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:40.117486 4407 consensus_queue.cc:237] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.119673 29322 catalog_manager.cc:5484] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:40.128134 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:40.132315 31990 raft_consensus.cc:1261] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:40.132403 32122 raft_consensus.cc:1261] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:40.132931 4174 consensus_queue.cc:1035] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.133438 2740 consensus_queue.cc:1035] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.137162 29322 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 58cbe304ca0c4052a959b5840be3394b (table rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b]). Expected version 1 got 0
I20240502 14:00:40.137539 29329 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 58cbe304ca0c4052a959b5840be3394b (table rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b]). Expected version 1 got 0
I20240502 14:00:40.140436 4423 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:40.140982 4423 tablet.cc:1620] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:40.142077 4424 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:40.142460 4424 tablet.cc:1620] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:40.145440 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 58cbe304ca0c4052a959b5840be3394b (table rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b]). Expected version 1 got 0
I20240502 14:00:40.145805 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b] alter complete (version 1)
I20240502 14:00:40.148208 4425 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:40.149317 4425 tablet.cc:1620] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:40.158428 32122 consensus_queue.cc:237] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:40.159726 29601 raft_consensus.cc:1261] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:40.159793 31860 raft_consensus.cc:1261] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:40.160216 3329 consensus_queue.cc:1035] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:40.160431 4236 consensus_queue.cc:1035] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:40.162003 3996 raft_consensus.cc:2932] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:40.162420 31860 raft_consensus.cc:2932] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:40.163525 29601 raft_consensus.cc:2932] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:40.164978 29320 catalog_manager.cc:5484] T 6d3a84c1d004412db381854147330dfc P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:40.169081 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d3a84c1d004412db381854147330dfc with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:40.169760 4439 tablet_replica.cc:330] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:40.169921 4439 raft_consensus.cc:2229] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:40.170071 4439 raft_consensus.cc:2258] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:40.170487 4439 ts_tablet_manager.cc:1875] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:40.174067 4439 ts_tablet_manager.cc:1888] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:40.174158 4439 log.cc:1196] T 6d3a84c1d004412db381854147330dfc P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/6d3a84c1d004412db381854147330dfc
I20240502 14:00:40.174918 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 6d3a84c1d004412db381854147330dfc (table rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]) successfully deleted
I20240502 14:00:40.249653 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:40.255502 4423 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:40.255841 4423 tablet.cc:1620] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:40.255964 4425 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:40.256341 4425 tablet.cc:1620] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:40.258239 4424 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:40.258543 4424 tablet.cc:1620] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:40.261773 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=24251d16337d4a1e8ef7ec58fdf5698b] alter complete (version 2)
I20240502 14:00:40.282649 3996 raft_consensus.cc:482] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:40.282790 3996 raft_consensus.cc:504] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.283325 3996 leader_election.cc:288] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:40.283717 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541806c11f074316a5342050427f4849" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:40.283807 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541806c11f074316a5342050427f4849" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:40.283869 31990 raft_consensus.cc:2454] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:40.283946 29471 raft_consensus.cc:2454] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:40.284282 32058 leader_election.cc:302] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.284540 3996 raft_consensus.cc:2781] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:40.284607 3996 raft_consensus.cc:482] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:40.284637 3996 raft_consensus.cc:3037] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.285646 3996 raft_consensus.cc:504] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.286000 3996 leader_election.cc:288] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:40.286361 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541806c11f074316a5342050427f4849" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:40.286499 31990 raft_consensus.cc:3037] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.286553 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541806c11f074316a5342050427f4849" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:40.286695 29471 raft_consensus.cc:3037] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.287540 31990 raft_consensus.cc:2454] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:40.287643 29471 raft_consensus.cc:2454] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:40.287832 32056 leader_election.cc:302] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.288086 3996 raft_consensus.cc:2781] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:40.288170 3996 raft_consensus.cc:686] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:40.288353 3996 consensus_queue.cc:237] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.290359 29325 catalog_manager.cc:5484] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:40.309861 31972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.327347 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:40.328449 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.335016 32235 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.335484 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.339444 31839 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:40.340039 29844 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:40.340804 32103 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:40.365435 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:40.371764 4423 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:40.375923 4424 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:40.383436 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=24251d16337d4a1e8ef7ec58fdf5698b] alter complete (version 3)
I20240502 14:00:40.389382 4425 tablet.cc:1722] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:40.425057 31990 consensus_queue.cc:237] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:40.426903 29471 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:40.427050 32255 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:40.427309 3575 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:40.427524 4505 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:40.429896 3575 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:40.431248 32255 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:40.431995 29471 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:40.432716 29325 catalog_manager.cc:5484] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:40.437393 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8e4ac081604ec8a3f463f740d9119c with delete_type TABLET_DATA_TOMBSTONED (TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:40.437793 4512 tablet_replica.cc:330] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:40.437948 4512 raft_consensus.cc:2229] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:40.438083 4512 raft_consensus.cc:2258] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:40.438692 4512 ts_tablet_manager.cc:1875] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:40.441479 4512 ts_tablet_manager.cc:1888] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:40.441574 4512 log.cc:1196] T ec8e4ac081604ec8a3f463f740d9119c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/ec8e4ac081604ec8a3f463f740d9119c
I20240502 14:00:40.442245 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet ec8e4ac081604ec8a3f463f740d9119c (table rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]) successfully deleted
I20240502 14:00:40.487973 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:40.488121 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:40.490234 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 58cbe304ca0c4052a959b5840be3394b
I20240502 14:00:40.491122 29844 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cbe304ca0c4052a959b5840be3394b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:40 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:40.491216 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cbe304ca0c4052a959b5840be3394b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:40 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:40.491248 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cbe304ca0c4052a959b5840be3394b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:40 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:40.491420 4512 tablet_replica.cc:330] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:40.491418 4439 tablet_replica.cc:330] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:40.491425 4551 tablet_replica.cc:330] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:40.491549 4551 raft_consensus.cc:2229] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:40.491552 4512 raft_consensus.cc:2229] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:40.491552 4439 raft_consensus.cc:2229] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:40.491667 4512 raft_consensus.cc:2258] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:40.491667 4439 raft_consensus.cc:2258] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:40.491695 4551 raft_consensus.cc:2258] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:40.492264 4551 ts_tablet_manager.cc:1875] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:40.492426 4512 ts_tablet_manager.cc:1875] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:40.492622 4439 ts_tablet_manager.cc:1875] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:40.494477 4551 ts_tablet_manager.cc:1888] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:40.494547 4551 log.cc:1196] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/58cbe304ca0c4052a959b5840be3394b
I20240502 14:00:40.494626 4439 ts_tablet_manager.cc:1888] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:40.494626 4512 ts_tablet_manager.cc:1888] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:40.494707 4439 log.cc:1196] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/58cbe304ca0c4052a959b5840be3394b
I20240502 14:00:40.494707 4512 log.cc:1196] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/58cbe304ca0c4052a959b5840be3394b
I20240502 14:00:40.494992 4551 ts_tablet_manager.cc:1909] T 58cbe304ca0c4052a959b5840be3394b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:40.495097 4512 ts_tablet_manager.cc:1909] T 58cbe304ca0c4052a959b5840be3394b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:40.495689 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 58cbe304ca0c4052a959b5840be3394b (table rebalancer_dynamic_table_new_name [id=24251d16337d4a1e8ef7ec58fdf5698b]) successfully deleted
I20240502 14:00:40.495759 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 58cbe304ca0c4052a959b5840be3394b (table rebalancer_dynamic_table_new_name [id=24251d16337d4a1e8ef7ec58fdf5698b]) successfully deleted
I20240502 14:00:40.500032 4407 consensus_queue.cc:1035] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.500701 4439 ts_tablet_manager.cc:1909] T 58cbe304ca0c4052a959b5840be3394b P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:40.502440 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 58cbe304ca0c4052a959b5840be3394b (table rebalancer_dynamic_table_new_name [id=24251d16337d4a1e8ef7ec58fdf5698b]) successfully deleted
I20240502 14:00:40.506208 4408 consensus_queue.cc:1035] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.558069 31972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.562093 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.571960 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.576463 4408 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.583768 4019 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.586252 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.587661 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:40.588586 32235 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.590474 32103 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:40.592737 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:40.601512 31839 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:40.607435 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 019a046637d54f578bb6d12fb1169544 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1]), partition=
I20240502 14:00:40.607721 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019a046637d54f578bb6d12fb1169544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.610141 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 019a046637d54f578bb6d12fb1169544 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1]), partition=
I20240502 14:00:40.610416 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019a046637d54f578bb6d12fb1169544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.612530 4571 tablet_bootstrap.cc:492] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:40.612717 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 019a046637d54f578bb6d12fb1169544 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1]), partition=
I20240502 14:00:40.612946 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019a046637d54f578bb6d12fb1169544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.613492 4571 tablet_bootstrap.cc:654] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.615123 4571 tablet_bootstrap.cc:492] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:40.615247 4571 ts_tablet_manager.cc:1367] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:00:40.615628 4573 tablet_bootstrap.cc:492] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:40.616554 4573 tablet_bootstrap.cc:654] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.617938 4573 tablet_bootstrap.cc:492] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:40.618016 4573 ts_tablet_manager.cc:1367] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:40.618428 4573 raft_consensus.cc:348] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.618528 4573 raft_consensus.cc:374] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.618556 4573 raft_consensus.cc:729] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.618672 4573 consensus_queue.cc:260] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.618959 4573 ts_tablet_manager.cc:1398] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:40.619585 4570 tablet_bootstrap.cc:492] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:40.622412 4570 tablet_bootstrap.cc:654] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.623209 4571 raft_consensus.cc:348] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.623401 4571 raft_consensus.cc:374] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.623471 4571 raft_consensus.cc:729] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.623665 4571 consensus_queue.cc:260] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.624084 4571 ts_tablet_manager.cc:1398] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240502 14:00:40.637670 4570 tablet_bootstrap.cc:492] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:40.637799 4570 ts_tablet_manager.cc:1367] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240502 14:00:40.638311 4570 raft_consensus.cc:348] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.638442 4570 raft_consensus.cc:374] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.638479 4570 raft_consensus.cc:729] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.638602 4570 consensus_queue.cc:260] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.638902 4570 ts_tablet_manager.cc:1398] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:40.656231 32122 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a1116d9f165b488796ab466f9a866471"
dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49162
I20240502 14:00:40.656407 32122 raft_consensus.cc:595] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:40.706810 3996 consensus_queue.cc:1035] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.713701 3329 consensus_queue.cc:1035] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.735381 4488 raft_consensus.cc:482] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:40.735513 4488 raft_consensus.cc:504] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.735895 4488 leader_election.cc:288] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:40.736696 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019a046637d54f578bb6d12fb1169544" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:40.736856 31860 raft_consensus.cc:2454] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:40.737046 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019a046637d54f578bb6d12fb1169544" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:40.737182 32122 raft_consensus.cc:2454] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:40.737854 29535 leader_election.cc:302] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:40.739435 4488 raft_consensus.cc:2781] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:40.739506 4488 raft_consensus.cc:482] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:40.739552 4488 raft_consensus.cc:3037] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.740666 4488 raft_consensus.cc:504] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.740983 4488 leader_election.cc:288] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:40.741581 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019a046637d54f578bb6d12fb1169544" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:40.741689 32122 raft_consensus.cc:3037] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.742631 32122 raft_consensus.cc:2454] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:40.743059 29535 leader_election.cc:302] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.743209 4488 raft_consensus.cc:2781] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:40.743268 4488 raft_consensus.cc:686] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:40.743388 4488 consensus_queue.cc:237] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:40.744287 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019a046637d54f578bb6d12fb1169544" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:40.744398 31860 raft_consensus.cc:3037] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.745357 31860 raft_consensus.cc:2454] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:40.749006 29329 catalog_manager.cc:5484] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:40.771533 29449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.785106 31972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.786317 29580 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:40.790879 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:40.791393 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.814411 32235 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:40.815723 31839 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:40.818272 32103 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:40.831689 3329 raft_consensus.cc:979] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: : Instructing follower 269b366a09ff435ea8ec8dc01d0f9f6e to start an election
I20240502 14:00:40.831811 3329 raft_consensus.cc:1067] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Signalling peer 269b366a09ff435ea8ec8dc01d0f9f6e to start an election
I20240502 14:00:40.832499 31990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a1116d9f165b488796ab466f9a866471"
dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
from {username='slave'} at 127.22.55.135:38501
I20240502 14:00:40.832671 31990 raft_consensus.cc:482] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:40.832723 31990 raft_consensus.cc:3037] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.833830 31990 raft_consensus.cc:504] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.834275 31990 leader_election.cc:288] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:40.835480 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116d9f165b488796ab466f9a866471" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:40.835608 32255 raft_consensus.cc:3037] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.835615 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116d9f165b488796ab466f9a866471" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:40.835747 29732 raft_consensus.cc:3037] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.835878 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116d9f165b488796ab466f9a866471" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:40.835995 32120 raft_consensus.cc:3032] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:40.836030 32120 raft_consensus.cc:729] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:40.836177 32120 consensus_queue.cc:260] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.836364 32120 raft_consensus.cc:3037] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.836702 29732 raft_consensus.cc:2454] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:00:40.836724 32255 raft_consensus.cc:2454] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:00:40.837317 32120 raft_consensus.cc:2454] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:00:40.837648 31926 leader_election.cc:302] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:40.837826 3575 raft_consensus.cc:2781] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:40.837893 3575 raft_consensus.cc:686] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:40.838033 3575 consensus_queue.cc:237] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.840520 29329 catalog_manager.cc:5484] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:40.847833 29329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:40.855950 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 053b20022c464c8b9e37bdd3c8ced56b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.856294 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 053b20022c464c8b9e37bdd3c8ced56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.856441 32103 tablet_service.cc:1467] Processing CreateTablet for tablet b23fc8a56f054a5b8ec0272ad99a93ea (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.856688 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23fc8a56f054a5b8ec0272ad99a93ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.856644 32233 tablet_service.cc:1467] Processing CreateTablet for tablet b23fc8a56f054a5b8ec0272ad99a93ea (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.856850 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23fc8a56f054a5b8ec0272ad99a93ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.858824 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 053b20022c464c8b9e37bdd3c8ced56b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.858924 4573 tablet_bootstrap.cc:492] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:40.858979 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 053b20022c464c8b9e37bdd3c8ced56b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.859103 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 053b20022c464c8b9e37bdd3c8ced56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.859222 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 053b20022c464c8b9e37bdd3c8ced56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.859900 4573 tablet_bootstrap.cc:654] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.861498 29449 tablet_service.cc:1467] Processing CreateTablet for tablet b23fc8a56f054a5b8ec0272ad99a93ea (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.861543 32103 tablet_service.cc:1467] Processing CreateTablet for tablet ea564959c91a43cb9cac2f58cece4de8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.861675 31971 tablet_service.cc:1467] Processing CreateTablet for tablet ea564959c91a43cb9cac2f58cece4de8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.861780 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea564959c91a43cb9cac2f58cece4de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.861814 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23fc8a56f054a5b8ec0272ad99a93ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.861889 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea564959c91a43cb9cac2f58cece4de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.861498 29451 tablet_service.cc:1467] Processing CreateTablet for tablet ea564959c91a43cb9cac2f58cece4de8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ca2dc5d543784ceaa797e06a28f07e83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:40.863317 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea564959c91a43cb9cac2f58cece4de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.864365 4573 tablet_bootstrap.cc:492] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:40.864457 4573 ts_tablet_manager.cc:1367] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 14:00:40.864840 4573 raft_consensus.cc:348] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.864917 4573 raft_consensus.cc:374] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.864938 4573 raft_consensus.cc:729] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.865027 4573 consensus_queue.cc:260] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.865228 4573 ts_tablet_manager.cc:1398] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:40.865374 4573 tablet_bootstrap.cc:492] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:40.866248 4573 tablet_bootstrap.cc:654] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.867228 4573 tablet_bootstrap.cc:492] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:40.867311 4573 ts_tablet_manager.cc:1367] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:40.867775 4573 raft_consensus.cc:348] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.867950 4573 raft_consensus.cc:374] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.868184 4637 tablet_bootstrap.cc:492] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:40.867986 4573 raft_consensus.cc:729] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.869664 4573 consensus_queue.cc:260] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.869982 4635 tablet_bootstrap.cc:492] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:40.870020 4573 ts_tablet_manager.cc:1398] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:40.871013 4637 tablet_bootstrap.cc:654] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.871399 4636 tablet_bootstrap.cc:492] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:40.872346 4636 tablet_bootstrap.cc:654] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.873721 4635 tablet_bootstrap.cc:654] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.873731 4636 tablet_bootstrap.cc:492] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:40.873971 4636 ts_tablet_manager.cc:1367] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:40.874213 4637 tablet_bootstrap.cc:492] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:40.874963 4637 ts_tablet_manager.cc:1367] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:40.874982 4633 tablet_bootstrap.cc:492] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:40.875285 4636 raft_consensus.cc:348] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.875450 4636 raft_consensus.cc:374] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.875519 4636 raft_consensus.cc:729] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.875682 4636 consensus_queue.cc:260] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.876085 4636 ts_tablet_manager.cc:1398] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:40.876444 4636 tablet_bootstrap.cc:492] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:40.876484 4635 tablet_bootstrap.cc:492] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:40.876646 4635 ts_tablet_manager.cc:1367] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:40.877092 4635 raft_consensus.cc:348] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.877337 4633 tablet_bootstrap.cc:654] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.877450 4636 tablet_bootstrap.cc:654] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.878011 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:40.878165 4635 raft_consensus.cc:374] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.878228 4635 raft_consensus.cc:729] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.878389 4635 consensus_queue.cc:260] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.878965 4635 ts_tablet_manager.cc:1398] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:40.879365 4637 raft_consensus.cc:348] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.879575 4637 raft_consensus.cc:374] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.879660 4637 raft_consensus.cc:729] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.879868 4637 consensus_queue.cc:260] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.879976 4633 tablet_bootstrap.cc:492] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:40.880086 4633 ts_tablet_manager.cc:1367] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:40.880331 4637 ts_tablet_manager.cc:1398] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:40.880565 4637 tablet_bootstrap.cc:492] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:40.880576 4633 raft_consensus.cc:348] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.880681 4633 raft_consensus.cc:374] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.880712 4633 raft_consensus.cc:729] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.880832 4633 consensus_queue.cc:260] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:40.881150 4633 ts_tablet_manager.cc:1398] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:40.881392 4633 tablet_bootstrap.cc:492] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:40.882339 4633 tablet_bootstrap.cc:654] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.882737 4636 tablet_bootstrap.cc:492] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:40.882812 4636 ts_tablet_manager.cc:1367] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:40.883317 4636 raft_consensus.cc:348] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.883496 4636 raft_consensus.cc:374] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.883571 4636 raft_consensus.cc:729] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.883759 4636 consensus_queue.cc:260] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.884151 4636 ts_tablet_manager.cc:1398] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:40.884965 4633 tablet_bootstrap.cc:492] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:40.885063 4633 ts_tablet_manager.cc:1367] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:40.885527 4633 raft_consensus.cc:348] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.885644 4633 raft_consensus.cc:374] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.885679 4633 raft_consensus.cc:729] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.885793 4633 consensus_queue.cc:260] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.886083 4633 ts_tablet_manager.cc:1398] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:40.887754 32120 raft_consensus.cc:1261] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:40.888254 31860 raft_consensus.cc:1261] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:40.888978 4488 consensus_queue.cc:1035] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.889170 2862 consensus_queue.cc:1035] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:40.891871 29326 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet 019a046637d54f578bb6d12fb1169544 (table rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1]). Expected version 1 got 0
I20240502 14:00:40.893246 29326 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 019a046637d54f578bb6d12fb1169544 (table rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1]). Expected version 1 got 0
I20240502 14:00:40.894374 4657 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:40.894774 4657 tablet.cc:1620] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:40.895556 4656 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:40.898484 4656 tablet.cc:1620] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:40.898988 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1] alter complete (version 1)
I20240502 14:00:40.899623 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 019a046637d54f578bb6d12fb1169544 (table rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1]). Expected version 1 got 0
I20240502 14:00:40.883873 4649 raft_consensus.cc:482] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:40.905413 4649 raft_consensus.cc:504] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.905839 4649 leader_election.cc:288] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:40.907258 4637 tablet_bootstrap.cc:654] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:40.907202 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea564959c91a43cb9cac2f58cece4de8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:40.907343 31990 raft_consensus.cc:2454] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:40.908126 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea564959c91a43cb9cac2f58cece4de8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:40.908265 32120 raft_consensus.cc:2454] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:40.907624 29404 leader_election.cc:302] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:40.909526 4649 raft_consensus.cc:2781] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:40.909611 4649 raft_consensus.cc:482] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:40.909651 4649 raft_consensus.cc:3037] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.910279 4658 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:40.910578 4658 tablet.cc:1620] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:40.910722 4649 raft_consensus.cc:504] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.911552 4649 leader_election.cc:288] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:40.911957 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea564959c91a43cb9cac2f58cece4de8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:40.911970 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea564959c91a43cb9cac2f58cece4de8" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:40.912074 32120 raft_consensus.cc:3037] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.912086 31990 raft_consensus.cc:3037] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.913004 32120 raft_consensus.cc:2454] T ea564959c91a43cb9cac2f58cece4de8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:40.913076 31990 raft_consensus.cc:2454] T ea564959c91a43cb9cac2f58cece4de8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:40.913323 29404 leader_election.cc:302] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.913893 4649 raft_consensus.cc:2781] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:40.913986 4649 raft_consensus.cc:686] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:40.914132 4649 consensus_queue.cc:237] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:40.915472 4637 tablet_bootstrap.cc:492] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:40.915616 4637 ts_tablet_manager.cc:1367] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240502 14:00:40.916141 4637 raft_consensus.cc:348] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.916252 29329 catalog_manager.cc:5484] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:40.916543 4637 raft_consensus.cc:374] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:40.916625 4637 raft_consensus.cc:729] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.916801 4637 consensus_queue.cc:260] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.917148 4637 ts_tablet_manager.cc:1398] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:40.929522 3329 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:40.929649 3329 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.930084 3329 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:40.930467 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:40.930543 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:40.930608 32255 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:40.930706 29471 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:40.932227 32058 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.932447 3329 raft_consensus.cc:2781] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:40.932519 3329 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:40.932566 3329 raft_consensus.cc:3037] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.933643 3329 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.933974 3329 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:40.934368 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:40.934478 29471 raft_consensus.cc:3037] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.935578 29471 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:40.935822 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:40.935925 32058 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:40.935966 32255 raft_consensus.cc:3037] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:40.936113 3329 raft_consensus.cc:2781] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:40.936204 3329 raft_consensus.cc:686] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:40.936328 3329 consensus_queue.cc:237] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:40.937031 32255 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:40.939786 29329 catalog_manager.cc:5484] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:41.003484 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:41.013593 4656 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:41.013928 4656 tablet.cc:1620] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:41.014618 4657 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:41.015208 4658 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:41.015435 4657 tablet.cc:1620] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:41.015511 4658 tablet.cc:1620] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:41.020457 2680 raft_consensus.cc:482] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:41.020588 2680 raft_consensus.cc:504] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:41.020998 2680 leader_election.cc:288] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:41.021323 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053b20022c464c8b9e37bdd3c8ced56b" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:41.021389 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053b20022c464c8b9e37bdd3c8ced56b" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:41.021461 31990 raft_consensus.cc:2454] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:41.021544 29732 raft_consensus.cc:2454] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:41.021772 32188 leader_election.cc:302] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:41.022059 2680 raft_consensus.cc:2781] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:41.022146 2680 raft_consensus.cc:482] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:41.022183 2680 raft_consensus.cc:3037] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.021996 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=7167a6658fd746e68e9281dd01aaceb1] alter complete (version 2)
I20240502 14:00:41.023218 2680 raft_consensus.cc:504] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:41.023608 2680 leader_election.cc:288] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:41.024335 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053b20022c464c8b9e37bdd3c8ced56b" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:41.024464 31990 raft_consensus.cc:3037] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.024638 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053b20022c464c8b9e37bdd3c8ced56b" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:41.024762 29732 raft_consensus.cc:3037] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.025554 31990 raft_consensus.cc:2454] T 053b20022c464c8b9e37bdd3c8ced56b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:41.025796 29732 raft_consensus.cc:2454] T 053b20022c464c8b9e37bdd3c8ced56b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:41.026290 32188 leader_election.cc:302] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:41.028334 2680 raft_consensus.cc:2781] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:41.028499 2680 raft_consensus.cc:686] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:41.028656 2680 consensus_queue.cc:237] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:41.031476 29327 catalog_manager.cc:5484] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:41.053442 31971 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.054647 29580 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:41.055928 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:41.056527 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.056761 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:41.057792 32235 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:41.072468 31839 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.093909 32103 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:41.128822 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:41.134902 4656 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:41.135824 4658 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:41.139129 4657 tablet.cc:1722] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:41.141863 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=7167a6658fd746e68e9281dd01aaceb1] alter complete (version 3)
I20240502 14:00:41.152670 29732 consensus_queue.cc:237] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:41.153889 31860 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:41.153914 29471 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:41.154304 4262 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:41.154531 4722 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:41.156427 4741 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:41.156766 29471 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:41.158036 31860 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:41.159504 29329 catalog_manager.cc:5484] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:41.162549 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc14b4bd42841df91972e0c5d02faa4 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:41.163161 4742 tablet_replica.cc:330] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:41.163364 4742 raft_consensus.cc:2229] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:41.163570 4742 raft_consensus.cc:2258] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:41.164110 4742 ts_tablet_manager.cc:1875] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:41.166395 4742 ts_tablet_manager.cc:1888] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:41.166476 4742 log.cc:1196] T 1fc14b4bd42841df91972e0c5d02faa4 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/1fc14b4bd42841df91972e0c5d02faa4
I20240502 14:00:41.167025 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 1fc14b4bd42841df91972e0c5d02faa4 (table rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]) successfully deleted
I20240502 14:00:41.222389 32122 consensus_queue.cc:237] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:41.223883 31990 raft_consensus.cc:1261] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:41.223883 32255 raft_consensus.cc:1261] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:41.224022 32058 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a34a1381cf84af6b560101d95447e6d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.224246 4789 consensus_queue.cc:1035] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:41.224428 3329 consensus_queue.cc:1035] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:41.226271 4789 raft_consensus.cc:2932] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:41.226680 32255 raft_consensus.cc:2932] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:41.228567 31990 raft_consensus.cc:2932] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:41.229785 29321 catalog_manager.cc:5484] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:41.243763 29327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:41.243939 29327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:41.246129 29327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 019a046637d54f578bb6d12fb1169544
I20240502 14:00:41.246913 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 019a046637d54f578bb6d12fb1169544 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:41 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:41.247155 4742 tablet_replica.cc:330] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:41.247176 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet 019a046637d54f578bb6d12fb1169544 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:41 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:41.247306 4742 raft_consensus.cc:2229] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:41.247471 4807 tablet_replica.cc:330] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:41.247509 4742 raft_consensus.cc:2258] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:41.247781 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet 019a046637d54f578bb6d12fb1169544 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:41 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:41.248075 4807 raft_consensus.cc:2229] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:41.248095 4808 tablet_replica.cc:330] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:41.248203 4742 ts_tablet_manager.cc:1875] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:41.248235 4808 raft_consensus.cc:2229] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:41.248284 4807 raft_consensus.cc:2258] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:41.248342 4808 raft_consensus.cc:2258] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:41.248940 4807 ts_tablet_manager.cc:1875] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:41.249322 4808 ts_tablet_manager.cc:1875] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:41.250564 4742 ts_tablet_manager.cc:1888] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:41.250630 4742 log.cc:1196] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/019a046637d54f578bb6d12fb1169544
I20240502 14:00:41.250943 4742 ts_tablet_manager.cc:1909] T 019a046637d54f578bb6d12fb1169544 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:41.251653 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 019a046637d54f578bb6d12fb1169544 (table rebalancer_dynamic_table_new_name [id=7167a6658fd746e68e9281dd01aaceb1]) successfully deleted
I20240502 14:00:41.251983 4808 ts_tablet_manager.cc:1888] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:41.252107 4808 log.cc:1196] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/019a046637d54f578bb6d12fb1169544
I20240502 14:00:41.252696 4808 ts_tablet_manager.cc:1909] T 019a046637d54f578bb6d12fb1169544 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:41.252923 31860 consensus_queue.cc:237] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:41.253062 4807 ts_tablet_manager.cc:1888] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:41.253118 4807 log.cc:1196] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/019a046637d54f578bb6d12fb1169544
I20240502 14:00:41.253418 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 019a046637d54f578bb6d12fb1169544 (table rebalancer_dynamic_table_new_name [id=7167a6658fd746e68e9281dd01aaceb1]) successfully deleted
I20240502 14:00:41.253543 4807 ts_tablet_manager.cc:1909] T 019a046637d54f578bb6d12fb1169544 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:41.254482 32255 raft_consensus.cc:1261] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:41.254541 31794 consensus_peers.cc:485] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0fa578318964569b2e10d7f858acd4d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.254817 4019 consensus_queue.cc:1035] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:41.254958 29862 raft_consensus.cc:1261] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:41.254981 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 019a046637d54f578bb6d12fb1169544 (table rebalancer_dynamic_table_new_name [id=7167a6658fd746e68e9281dd01aaceb1]) successfully deleted
I20240502 14:00:41.255453 4019 consensus_queue.cc:1035] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:41.257052 4407 raft_consensus.cc:2932] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:41.257486 29862 raft_consensus.cc:2932] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:41.258713 32255 raft_consensus.cc:2932] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:41.260286 29327 catalog_manager.cc:5484] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:41.281535 31860 consensus_queue.cc:237] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
W20240502 14:00:41.283433 31795 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df1c6786a80e40a98130a5d9057e2877. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.283778 32120 raft_consensus.cc:1261] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:41.284152 4812 consensus_queue.cc:1035] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:41.284276 31990 raft_consensus.cc:1261] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:41.284694 4812 consensus_queue.cc:1035] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:41.286428 4019 raft_consensus.cc:2932] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:41.286801 31990 raft_consensus.cc:2932] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:41.290824 32120 raft_consensus.cc:2932] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:41.291188 29325 catalog_manager.cc:5484] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:41.297318 29732 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:00:41.297740 4505 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:41.318526 29470 consensus_queue.cc:237] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:41.319924 29862 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:41.320034 29404 consensus_peers.cc:485] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f1a1a1592e84b3898a06d4360bac846. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.320241 29732 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:41.320591 4649 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:41.320761 4096 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:41.323000 4096 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:41.325400 29732 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:41.325690 29862 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:41.325938 29321 catalog_manager.cc:5484] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:41.347985 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:41.352603 29470 consensus_queue.cc:237] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:41.355152 32235 tablet_service.cc:1467] Processing CreateTablet for tablet d0e476c1615c4bce95210c6566c3f7f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a]), partition=
I20240502 14:00:41.355252 32103 tablet_service.cc:1467] Processing CreateTablet for tablet d0e476c1615c4bce95210c6566c3f7f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a]), partition=
I20240502 14:00:41.355458 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e476c1615c4bce95210c6566c3f7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.355525 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e476c1615c4bce95210c6566c3f7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.357301 4573 tablet_bootstrap.cc:492] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:41.357551 31839 tablet_service.cc:1467] Processing CreateTablet for tablet d0e476c1615c4bce95210c6566c3f7f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a]), partition=
I20240502 14:00:41.357692 4633 tablet_bootstrap.cc:492] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:41.357791 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e476c1615c4bce95210c6566c3f7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.358287 4573 tablet_bootstrap.cc:654] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.358490 4633 tablet_bootstrap.cc:654] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.360862 29862 raft_consensus.cc:1261] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:41.361405 4633 tablet_bootstrap.cc:492] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:41.361405 4651 consensus_queue.cc:1035] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:41.361506 4633 ts_tablet_manager.cc:1367] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:41.361719 29601 raft_consensus.cc:1261] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:41.362015 4633 raft_consensus.cc:348] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.362130 4633 raft_consensus.cc:374] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.362123 4651 consensus_queue.cc:1035] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:41.362186 4633 raft_consensus.cc:729] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
W20240502 14:00:41.362300 29406 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7cc49f0227d491ba7a8df281856b8f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.362304 4633 consensus_queue.cc:260] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.362620 4633 ts_tablet_manager.cc:1398] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.362809 4871 tablet_bootstrap.cc:492] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:41.363874 4871 tablet_bootstrap.cc:654] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.364138 4573 tablet_bootstrap.cc:492] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:41.364234 4573 ts_tablet_manager.cc:1367] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:41.364619 4573 raft_consensus.cc:348] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.364724 4573 raft_consensus.cc:374] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.364756 4573 raft_consensus.cc:729] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.364899 4573 consensus_queue.cc:260] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.365182 4573 ts_tablet_manager.cc:1398] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:41.365281 4871 tablet_bootstrap.cc:492] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:41.365356 4871 ts_tablet_manager.cc:1367] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:41.366276 4651 raft_consensus.cc:2932] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:41.365867 4871 raft_consensus.cc:348] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.367285 4871 raft_consensus.cc:374] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.367341 4871 raft_consensus.cc:729] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.367482 4871 consensus_queue.cc:260] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.367733 4871 ts_tablet_manager.cc:1398] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:00:41.368556 29862 raft_consensus.cc:2932] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:41.368948 29601 raft_consensus.cc:2932] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:41.373291 29322 catalog_manager.cc:5484] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:41.407153 2680 consensus_queue.cc:1035] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.410364 4709 consensus_queue.cc:1035] T 053b20022c464c8b9e37bdd3c8ced56b P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.430518 4789 consensus_queue.cc:1035] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.430675 32120 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:00:41.431249 4505 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:41.434427 32255 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:00:41.435427 4505 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:41.440284 4789 consensus_queue.cc:1035] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.440899 4651 consensus_queue.cc:1035] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.445953 4851 consensus_queue.cc:1035] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.473351 4709 raft_consensus.cc:482] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:41.473477 4709 raft_consensus.cc:504] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.473865 4709 leader_election.cc:288] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:41.474772 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e476c1615c4bce95210c6566c3f7f8" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:41.474968 31860 raft_consensus.cc:2454] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:41.475219 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e476c1615c4bce95210c6566c3f7f8" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:41.475340 32120 raft_consensus.cc:2454] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:41.476541 32188 leader_election.cc:302] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:41.476912 2680 raft_consensus.cc:2781] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:41.476969 2680 raft_consensus.cc:482] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:41.477005 2680 raft_consensus.cc:3037] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.478312 2680 raft_consensus.cc:504] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.478732 2680 leader_election.cc:288] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:41.480489 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e476c1615c4bce95210c6566c3f7f8" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:41.480630 32122 raft_consensus.cc:3037] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.481889 32122 raft_consensus.cc:2454] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:41.483814 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e476c1615c4bce95210c6566c3f7f8" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:41.483969 31860 raft_consensus.cc:3037] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.485220 31860 raft_consensus.cc:2454] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:41.488483 32188 leader_election.cc:302] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:41.489631 2680 raft_consensus.cc:2781] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:41.489740 2680 raft_consensus.cc:686] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:41.489908 2680 consensus_queue.cc:237] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.493942 29321 catalog_manager.cc:5484] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:41.515187 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.524742 29844 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:41.526948 32235 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:41.530153 31971 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.535650 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:41.536245 29580 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:41.546108 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:41.555436 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 14ea89aafe48423491f90865328ced42 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.555785 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ea89aafe48423491f90865328ced42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.557407 31839 tablet_service.cc:1467] Processing CreateTablet for tablet a113227860bc48078340ec76873be7e1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.557407 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 14ea89aafe48423491f90865328ced42 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.557680 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a113227860bc48078340ec76873be7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.557680 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ea89aafe48423491f90865328ced42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.558259 29844 tablet_service.cc:1467] Processing CreateTablet for tablet 14ea89aafe48423491f90865328ced42 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.558535 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14ea89aafe48423491f90865328ced42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.559258 29582 tablet_service.cc:1467] Processing CreateTablet for tablet a113227860bc48078340ec76873be7e1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.559509 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a113227860bc48078340ec76873be7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.560077 29842 tablet_service.cc:1467] Processing CreateTablet for tablet a113227860bc48078340ec76873be7e1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.560318 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a113227860bc48078340ec76873be7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.560796 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 31df5c53f5004ef399a1bd3458f57573 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.560992 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31df5c53f5004ef399a1bd3458f57573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.561709 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 31df5c53f5004ef399a1bd3458f57573 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.561962 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31df5c53f5004ef399a1bd3458f57573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.562965 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 31df5c53f5004ef399a1bd3458f57573 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=ccff20fdd13e46da91df63b9f1f2a8f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:41.563247 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31df5c53f5004ef399a1bd3458f57573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.565116 4573 tablet_bootstrap.cc:492] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:41.565341 31840 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.565557 4633 tablet_bootstrap.cc:492] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:41.566066 4573 tablet_bootstrap.cc:654] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.566449 4633 tablet_bootstrap.cc:654] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.568073 4871 tablet_bootstrap.cc:492] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:41.569018 4871 tablet_bootstrap.cc:654] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.571483 4954 tablet_bootstrap.cc:492] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:41.572402 4633 tablet_bootstrap.cc:492] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:41.572496 4633 ts_tablet_manager.cc:1367] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:41.572983 4633 raft_consensus.cc:348] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.573092 4633 raft_consensus.cc:374] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.573124 4633 raft_consensus.cc:729] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.573239 4633 consensus_queue.cc:260] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.573552 4633 ts_tablet_manager.cc:1398] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:41.573626 4955 tablet_bootstrap.cc:492] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:41.574537 4953 tablet_bootstrap.cc:492] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:41.574602 4955 tablet_bootstrap.cc:654] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.575160 4871 tablet_bootstrap.cc:492] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:41.575258 4871 ts_tablet_manager.cc:1367] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:41.575696 4871 raft_consensus.cc:348] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.575811 4871 raft_consensus.cc:374] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.575840 4871 raft_consensus.cc:729] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.575946 4871 consensus_queue.cc:260] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.576174 4871 ts_tablet_manager.cc:1398] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.576972 4573 tablet_bootstrap.cc:492] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:41.577071 4573 ts_tablet_manager.cc:1367] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240502 14:00:41.577466 4953 tablet_bootstrap.cc:654] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.577543 4573 raft_consensus.cc:348] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.577656 4573 raft_consensus.cc:374] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.577690 4573 raft_consensus.cc:729] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.577821 4573 consensus_queue.cc:260] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.578153 4573 ts_tablet_manager.cc:1398] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:41.578168 4954 tablet_bootstrap.cc:654] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.579712 4954 tablet_bootstrap.cc:492] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:41.579842 4954 ts_tablet_manager.cc:1367] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:41.580317 4954 raft_consensus.cc:348] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.580655 4954 raft_consensus.cc:374] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.580741 4954 raft_consensus.cc:729] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.580906 4954 consensus_queue.cc:260] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.581946 4954 ts_tablet_manager.cc:1398] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:41.587297 4955 tablet_bootstrap.cc:492] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:41.588274 4955 ts_tablet_manager.cc:1367] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240502 14:00:41.596904 4955 raft_consensus.cc:348] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.597123 4955 raft_consensus.cc:374] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.597199 4955 raft_consensus.cc:729] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.597371 4955 consensus_queue.cc:260] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.598839 4955 ts_tablet_manager.cc:1398] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240502 14:00:41.599083 4955 tablet_bootstrap.cc:492] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:41.599646 32103 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:41.601374 4789 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:41.601481 4789 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.601879 4789 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:41.602283 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:41.602512 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:41.602648 32254 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
W20240502 14:00:41.602705 32057 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:41.602933 32058 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:41.603217 4789 raft_consensus.cc:2781] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:41.603271 4789 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:41.603303 4789 raft_consensus.cc:3037] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.604382 4789 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.604745 4789 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:41.604799 4953 tablet_bootstrap.cc:492] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:41.604887 4953 ts_tablet_manager.cc:1367] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 14:00:41.605199 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:41.605405 4953 raft_consensus.cc:348] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.605504 4953 raft_consensus.cc:374] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.605530 4953 raft_consensus.cc:729] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
W20240502 14:00:41.605515 32057 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:41.605640 4953 consensus_queue.cc:260] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.605888 4953 ts_tablet_manager.cc:1398] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.606035 4953 tablet_bootstrap.cc:492] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:41.607005 4953 tablet_bootstrap.cc:654] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.607002 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:41.607127 32254 raft_consensus.cc:3037] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.608170 32254 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:41.608512 32058 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:41.608664 4789 raft_consensus.cc:2781] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:41.608808 4789 raft_consensus.cc:686] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:41.608924 4789 consensus_queue.cc:237] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.611445 29320 catalog_manager.cc:5484] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:41.609781 4955 tablet_bootstrap.cc:654] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.616050 4955 tablet_bootstrap.cc:492] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:41.616220 4955 ts_tablet_manager.cc:1367] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:41.616709 4955 raft_consensus.cc:348] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.616884 4955 raft_consensus.cc:374] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.617537 4955 raft_consensus.cc:729] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.617564 4965 ts_tablet_manager.cc:902] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:41.617736 4955 consensus_queue.cc:260] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.618309 4955 ts_tablet_manager.cc:1398] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:41.618606 4955 tablet_bootstrap.cc:492] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:41.620401 4955 tablet_bootstrap.cc:654] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:41.621616 4955 tablet_bootstrap.cc:492] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:41.621690 4955 ts_tablet_manager.cc:1367] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:41.622077 4955 raft_consensus.cc:348] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.622169 4955 raft_consensus.cc:374] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.622195 4955 raft_consensus.cc:729] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.622319 4955 consensus_queue.cc:260] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.622598 4955 ts_tablet_manager.cc:1398] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:41.622586 4965 tablet_copy_client.cc:308] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:41.623255 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 3a34a1381cf84af6b560101d95447e6d from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:32873)
I20240502 14:00:41.623378 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 3a34a1381cf84af6b560101d95447e6d from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:32873: session id = 1d00d3a47c52478894cf8c66959ef7bd-3a34a1381cf84af6b560101d95447e6d
I20240502 14:00:41.624408 32143 tablet_copy_source_session.cc:205] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.625795 4953 tablet_bootstrap.cc:492] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:41.625885 4953 ts_tablet_manager.cc:1367] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240502 14:00:41.626456 4965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a34a1381cf84af6b560101d95447e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.626924 4953 raft_consensus.cc:348] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.627039 4953 raft_consensus.cc:374] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:41.627066 4953 raft_consensus.cc:729] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.627188 4953 consensus_queue.cc:260] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.628093 4953 ts_tablet_manager.cc:1398] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:41.628832 4965 tablet_copy_client.cc:783] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.628975 4965 tablet_copy_client.cc:647] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.629782 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:41.634910 32122 raft_consensus.cc:1261] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:41.634990 31860 raft_consensus.cc:1261] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:41.635556 2680 consensus_queue.cc:1035] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.635869 2680 consensus_queue.cc:1035] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.639469 4965 tablet_copy_client.cc:523] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:41.639482 29321 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet d0e476c1615c4bce95210c6566c3f7f8 (table rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a]). Expected version 1 got 0
I20240502 14:00:41.640677 29320 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet d0e476c1615c4bce95210c6566c3f7f8 (table rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a]). Expected version 1 got 0
I20240502 14:00:41.642647 4965 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:41.646116 4892 mvcc.cc:205] Tried to move back new op lower bound from 7023240976931188736 to 7023240976344547328. Current Snapshot: MvccSnapshot[applied={T|T < 7023240976931188736}]
I20240502 14:00:41.646908 4976 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:41.647351 4976 tablet.cc:1620] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:41.648485 29322 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet d0e476c1615c4bce95210c6566c3f7f8 (table rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a]). Expected version 1 got 0
I20240502 14:00:41.649011 4965 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:41.649405 4965 tablet_bootstrap.cc:492] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:00:41.649637 4965 ts_tablet_manager.cc:1367] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:00:41.650209 4965 raft_consensus.cc:348] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:41.650420 4965 raft_consensus.cc:729] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.650446 29321 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet d0e476c1615c4bce95210c6566c3f7f8 (table rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a]). Expected version 1 got 0
I20240502 14:00:41.650700 4965 consensus_queue.cc:260] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:41.650928 4977 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:41.651178 4965 ts_tablet_manager.cc:1398] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.651561 4977 tablet.cc:1620] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:41.651804 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-3a34a1381cf84af6b560101d95447e6d received from {username='slave'} at 127.22.55.129:32873
I20240502 14:00:41.651883 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-3a34a1381cf84af6b560101d95447e6d on tablet 3a34a1381cf84af6b560101d95447e6d with peer 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:41.654994 4975 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:41.655387 4975 tablet.cc:1620] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:41.656698 29321 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a] alter complete (version 1)
I20240502 14:00:41.658546 4984 ts_tablet_manager.cc:902] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:41.659986 4984 tablet_copy_client.cc:308] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:00:41.660478 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet b0fa578318964569b2e10d7f858acd4d from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:42227)
I20240502 14:00:41.660573 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet b0fa578318964569b2e10d7f858acd4d from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:42227: session id = d6e2e2c10def4331bec17bd110c9b423-b0fa578318964569b2e10d7f858acd4d
I20240502 14:00:41.661566 31865 tablet_copy_source_session.cc:205] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.662379 4984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0fa578318964569b2e10d7f858acd4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.664486 4984 tablet_copy_client.cc:783] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.664651 4984 tablet_copy_client.cc:647] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.665740 4984 tablet_copy_client.cc:523] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:41.667111 4984 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:41.670550 4984 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:41.670776 4984 tablet_bootstrap.cc:492] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:41.670961 4984 ts_tablet_manager.cc:1367] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:00:41.671430 4984 raft_consensus.cc:348] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:41.671568 4984 raft_consensus.cc:729] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.671711 4984 consensus_queue.cc:260] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:41.671978 4984 ts_tablet_manager.cc:1398] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.672535 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-b0fa578318964569b2e10d7f858acd4d received from {username='slave'} at 127.22.55.130:42227
I20240502 14:00:41.672616 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-b0fa578318964569b2e10d7f858acd4d on tablet b0fa578318964569b2e10d7f858acd4d with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:41.683811 4174 raft_consensus.cc:482] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:41.683979 4174 raft_consensus.cc:504] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.684406 4174 leader_election.cc:288] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:41.684839 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ea89aafe48423491f90865328ced42" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:41.684991 29601 raft_consensus.cc:2454] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:41.685032 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ea89aafe48423491f90865328ced42" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:41.685231 31990 raft_consensus.cc:2454] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:41.685334 29797 leader_election.cc:302] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:41.685644 4174 raft_consensus.cc:2781] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:41.685717 4174 raft_consensus.cc:482] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:41.685766 4174 raft_consensus.cc:3037] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.686982 4174 raft_consensus.cc:504] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.687322 4174 leader_election.cc:288] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:41.687637 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ea89aafe48423491f90865328ced42" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:41.687671 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14ea89aafe48423491f90865328ced42" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:41.687767 29601 raft_consensus.cc:3037] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.687767 31990 raft_consensus.cc:3037] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.688760 31990 raft_consensus.cc:2454] T 14ea89aafe48423491f90865328ced42 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:41.688759 29601 raft_consensus.cc:2454] T 14ea89aafe48423491f90865328ced42 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:41.689110 29797 leader_election.cc:302] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:41.689400 4174 raft_consensus.cc:2781] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:41.689494 4174 raft_consensus.cc:686] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:41.689620 4174 consensus_queue.cc:237] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.691726 29322 catalog_manager.cc:5484] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:41.711019 4407 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:41.711141 4407 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.711591 4407 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:41.712399 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a113227860bc48078340ec76873be7e1" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:41.712548 29601 raft_consensus.cc:2454] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:41.712930 31794 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:41.713202 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a113227860bc48078340ec76873be7e1" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:41.713333 29862 raft_consensus.cc:2454] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:41.713788 4407 raft_consensus.cc:2781] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:41.713845 4407 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:41.713883 4407 raft_consensus.cc:3037] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.715029 4407 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.715374 4407 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:41.716701 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a113227860bc48078340ec76873be7e1" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:41.716809 29601 raft_consensus.cc:3037] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.717747 29601 raft_consensus.cc:2454] T a113227860bc48078340ec76873be7e1 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:41.719446 31794 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:41.720252 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a113227860bc48078340ec76873be7e1" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:41.720381 29862 raft_consensus.cc:3037] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:41.721385 29862 raft_consensus.cc:2454] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:41.722193 4407 raft_consensus.cc:2781] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:41.722294 4407 raft_consensus.cc:686] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:41.722441 4407 consensus_queue.cc:237] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:41.724676 29323 catalog_manager.cc:5484] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:41.760694 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:41.768623 4975 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:41.769009 4975 tablet.cc:1620] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:41.770015 4977 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:41.770337 4977 tablet.cc:1620] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:41.771149 4976 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:41.771456 4976 tablet.cc:1620] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:41.776175 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=ead25c8ca1c148858359c53843376e3a] alter complete (version 2)
I20240502 14:00:41.792052 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.793707 31971 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:41.798770 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:41.804970 5039 ts_tablet_manager.cc:902] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:41.806466 32235 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:41.805577 5039 tablet_copy_client.cc:308] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:41.811337 31839 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:41.812965 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet a7cc49f0227d491ba7a8df281856b8f8 from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57511)
I20240502 14:00:41.813081 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet a7cc49f0227d491ba7a8df281856b8f8 from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57511: session id = 8973f2527acb48f5ad36589652de73e1-a7cc49f0227d491ba7a8df281856b8f8
I20240502 14:00:41.814105 29491 tablet_copy_source_session.cc:205] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.815452 5039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7cc49f0227d491ba7a8df281856b8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.817593 32103 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:41.823941 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:41.825676 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:41.830178 5039 tablet_copy_client.cc:783] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.830403 5039 tablet_copy_client.cc:647] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.835680 5039 tablet_copy_client.cc:523] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:41.838636 5039 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:41.863400 5039 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:41.863818 5039 tablet_bootstrap.cc:492] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:41.864007 5039 ts_tablet_manager.cc:1367] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20240502 14:00:41.864434 5039 raft_consensus.cc:348] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:41.864570 5039 raft_consensus.cc:729] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.864734 5039 consensus_queue.cc:260] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:41.865159 5039 ts_tablet_manager.cc:1398] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.865779 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-a7cc49f0227d491ba7a8df281856b8f8 received from {username='slave'} at 127.22.55.136:57511
I20240502 14:00:41.865859 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-a7cc49f0227d491ba7a8df281856b8f8 on tablet a7cc49f0227d491ba7a8df281856b8f8 with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:41.871049 5045 ts_tablet_manager.cc:902] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:41.873739 5045 tablet_copy_client.cc:308] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:00:41.874300 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet df1c6786a80e40a98130a5d9057e2877 from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:35643)
I20240502 14:00:41.874420 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet df1c6786a80e40a98130a5d9057e2877 from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:35643: session id = efdbf7bdfe0b41bdadb1add053b09a7e-df1c6786a80e40a98130a5d9057e2877
I20240502 14:00:41.875442 31865 tablet_copy_source_session.cc:205] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.876350 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1c6786a80e40a98130a5d9057e2877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.878677 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:41.878751 5045 tablet_copy_client.cc:783] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.878937 5045 tablet_copy_client.cc:647] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.886773 5045 tablet_copy_client.cc:523] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:41.887907 4975 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:41.888774 5045 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:41.892866 4977 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:41.892904 4976 tablet.cc:1722] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:41.893491 5045 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:41.893841 29324 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=ead25c8ca1c148858359c53843376e3a] alter complete (version 3)
I20240502 14:00:41.893878 5045 tablet_bootstrap.cc:492] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:00:41.894037 5045 ts_tablet_manager.cc:1367] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20240502 14:00:41.894788 5045 raft_consensus.cc:348] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:41.894960 5045 raft_consensus.cc:729] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.895128 5045 consensus_queue.cc:260] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:41.895526 5045 ts_tablet_manager.cc:1398] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:41.897511 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-df1c6786a80e40a98130a5d9057e2877 received from {username='slave'} at 127.22.55.131:35643
I20240502 14:00:41.897590 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-df1c6786a80e40a98130a5d9057e2877 on tablet df1c6786a80e40a98130a5d9057e2877 with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:41.914340 5062 ts_tablet_manager.cc:902] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:41.914986 5062 tablet_copy_client.cc:308] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:41.918701 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet 6f1a1a1592e84b3898a06d4360bac846 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:45577)
I20240502 14:00:41.918828 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet 6f1a1a1592e84b3898a06d4360bac846 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:45577: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-6f1a1a1592e84b3898a06d4360bac846
I20240502 14:00:41.919931 29491 tablet_copy_source_session.cc:205] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.921202 5062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1a1a1592e84b3898a06d4360bac846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.923616 5062 tablet_copy_client.cc:783] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.923853 5062 tablet_copy_client.cc:647] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.925020 5062 tablet_copy_client.cc:523] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:41.926429 5062 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:41.952711 5062 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:41.953147 5062 tablet_bootstrap.cc:492] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:41.953382 5062 ts_tablet_manager.cc:1367] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.002s
I20240502 14:00:41.953840 5062 raft_consensus.cc:348] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:41.954026 5062 raft_consensus.cc:729] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:41.954229 5062 consensus_queue.cc:260] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:41.955797 5062 ts_tablet_manager.cc:1398] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:41.956290 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-6f1a1a1592e84b3898a06d4360bac846 received from {username='slave'} at 127.22.55.134:45577
I20240502 14:00:41.956386 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-6f1a1a1592e84b3898a06d4360bac846 on tablet 6f1a1a1592e84b3898a06d4360bac846 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:41.995551 29470 raft_consensus.cc:1203] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:41.996865 4174 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:41.996975 4174 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:41.997390 4174 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:41.998313 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:41.998498 32255 raft_consensus.cc:2379] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in current term 1: Already voted for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in this term.
I20240502 14:00:41.998973 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:41.999596 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:41.999756 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:42.000207 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:42.000371 4174 raft_consensus.cc:3037] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.002928 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet d0e476c1615c4bce95210c6566c3f7f8
I20240502 14:00:42.003042 4174 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:00:42.003546 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e476c1615c4bce95210c6566c3f7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:41 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:42.004098 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e476c1615c4bce95210c6566c3f7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:41 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:42.004094 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e476c1615c4bce95210c6566c3f7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:41 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:42.004525 5100 tablet_replica.cc:330] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:42.004690 5100 raft_consensus.cc:2229] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:42.004915 5100 raft_consensus.cc:2258] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.005154 5101 tablet_replica.cc:330] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:42.005369 5101 raft_consensus.cc:2229] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:42.005513 5101 raft_consensus.cc:2258] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.005506 5100 ts_tablet_manager.cc:1875] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:42.007813 5100 ts_tablet_manager.cc:1888] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:42.007891 5100 log.cc:1196] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/d0e476c1615c4bce95210c6566c3f7f8
I20240502 14:00:42.007889 5101 ts_tablet_manager.cc:1875] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:42.009078 3996 consensus_queue.cc:1035] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.009690 5100 ts_tablet_manager.cc:1909] T d0e476c1615c4bce95210c6566c3f7f8 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:42.010624 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet d0e476c1615c4bce95210c6566c3f7f8 (table rebalancer_dynamic_table_new_name [id=ead25c8ca1c148858359c53843376e3a]) successfully deleted
I20240502 14:00:42.010900 5101 ts_tablet_manager.cc:1888] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:42.010982 5101 log.cc:1196] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/d0e476c1615c4bce95210c6566c3f7f8
I20240502 14:00:42.011241 5099 tablet_replica.cc:330] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:42.011330 5101 ts_tablet_manager.cc:1909] T d0e476c1615c4bce95210c6566c3f7f8 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:42.011468 5099 raft_consensus.cc:2229] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:42.011624 5099 raft_consensus.cc:2258] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.012221 5099 ts_tablet_manager.cc:1875] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:42.013178 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet d0e476c1615c4bce95210c6566c3f7f8 (table rebalancer_dynamic_table_new_name [id=ead25c8ca1c148858359c53843376e3a]) successfully deleted
I20240502 14:00:42.014707 5099 ts_tablet_manager.cc:1888] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:42.014832 5099 log.cc:1196] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/d0e476c1615c4bce95210c6566c3f7f8
I20240502 14:00:42.015636 5099 ts_tablet_manager.cc:1909] T d0e476c1615c4bce95210c6566c3f7f8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:42.017251 4789 consensus_queue.cc:1035] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.017894 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet d0e476c1615c4bce95210c6566c3f7f8 (table rebalancer_dynamic_table_new_name [id=ead25c8ca1c148858359c53843376e3a]) successfully deleted
I20240502 14:00:42.032935 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:42.039907 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:42.040362 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:42.040987 31971 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:42.043833 32235 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:42.045477 31839 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:42.059342 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:42.081427 32103 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:42.092048 29601 raft_consensus.cc:1203] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:42.106693 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:42.113662 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 4908435d41404a12bc145f39b32a64e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5]), partition=
I20240502 14:00:42.113794 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 4908435d41404a12bc145f39b32a64e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5]), partition=
I20240502 14:00:42.113934 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4908435d41404a12bc145f39b32a64e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.114001 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4908435d41404a12bc145f39b32a64e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.116250 5117 tablet_bootstrap.cc:492] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:42.116503 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 4908435d41404a12bc145f39b32a64e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5]), partition=
I20240502 14:00:42.116748 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4908435d41404a12bc145f39b32a64e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.117390 5117 tablet_bootstrap.cc:654] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.117964 5118 tablet_bootstrap.cc:492] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:42.118485 5117 tablet_bootstrap.cc:492] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:42.118554 4953 tablet_bootstrap.cc:492] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:42.118583 5117 ts_tablet_manager.cc:1367] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:42.119128 5118 tablet_bootstrap.cc:654] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.119375 4953 tablet_bootstrap.cc:654] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.120317 5117 raft_consensus.cc:348] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.120523 5117 raft_consensus.cc:374] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.120605 5117 raft_consensus.cc:729] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.120736 5117 consensus_queue.cc:260] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.121556 5118 tablet_bootstrap.cc:492] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:42.121572 4953 tablet_bootstrap.cc:492] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:42.121649 5118 ts_tablet_manager.cc:1367] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:42.121649 4953 ts_tablet_manager.cc:1367] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:42.122119 5118 raft_consensus.cc:348] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.122244 5118 raft_consensus.cc:374] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.122280 5118 raft_consensus.cc:729] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.122238 4953 raft_consensus.cc:348] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.122351 4953 raft_consensus.cc:374] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.122390 4953 raft_consensus.cc:729] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.122399 5118 consensus_queue.cc:260] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.122485 4953 consensus_queue.cc:260] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.122684 5118 ts_tablet_manager.cc:1398] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.122735 4953 ts_tablet_manager.cc:1398] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.131619 4407 consensus_queue.cc:1035] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.135362 5117 ts_tablet_manager.cc:1398] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:42.135712 5135 consensus_queue.cc:1035] T a113227860bc48078340ec76873be7e1 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.148447 4831 raft_consensus.cc:482] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.148566 4831 raft_consensus.cc:504] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.148938 4831 leader_election.cc:288] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:42.149288 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4908435d41404a12bc145f39b32a64e0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:42.149453 32120 raft_consensus.cc:2454] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:42.149498 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4908435d41404a12bc145f39b32a64e0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:42.149619 29600 raft_consensus.cc:2454] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:42.149724 31794 leader_election.cc:302] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:42.150357 4831 raft_consensus.cc:2781] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:42.150413 4831 raft_consensus.cc:482] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:42.150447 4831 raft_consensus.cc:3037] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.151517 4831 raft_consensus.cc:504] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.151799 4831 leader_election.cc:288] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:42.152266 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4908435d41404a12bc145f39b32a64e0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:42.152405 29601 raft_consensus.cc:3037] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.152943 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4908435d41404a12bc145f39b32a64e0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:42.153050 32120 raft_consensus.cc:3037] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.153504 29601 raft_consensus.cc:2454] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:42.154048 32120 raft_consensus.cc:2454] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:42.154733 31794 leader_election.cc:302] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:42.154920 4831 raft_consensus.cc:2781] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:42.155017 4831 raft_consensus.cc:686] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:42.155162 4831 consensus_queue.cc:237] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.157383 29325 catalog_manager.cc:5484] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:42.202932 4174 consensus_queue.cc:1035] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.207567 3671 consensus_queue.cc:1035] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.236274 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:42.237998 31971 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:42.238627 29324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:42.246905 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 1203440714974cb5981e67c2856fb0c7 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.247229 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1203440714974cb5981e67c2856fb0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.247311 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 1203440714974cb5981e67c2856fb0c7 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.247387 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 2a41450335344abe9d261d6a00674293 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.247563 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1203440714974cb5981e67c2856fb0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.247673 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a41450335344abe9d261d6a00674293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.249604 4953 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:42.250613 4953 tablet_bootstrap.cc:654] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.250895 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 2a41450335344abe9d261d6a00674293 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.250888 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 1203440714974cb5981e67c2856fb0c7 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.251222 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1203440714974cb5981e67c2856fb0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.251405 32234 tablet_service.cc:1467] Processing CreateTablet for tablet f787b154dfd44dfe9cf1ef2f996bfffb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.251637 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f787b154dfd44dfe9cf1ef2f996bfffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.253319 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a41450335344abe9d261d6a00674293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.253845 5187 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:42.254739 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 2a41450335344abe9d261d6a00674293 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.254890 5187 tablet_bootstrap.cc:654] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.255019 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a41450335344abe9d261d6a00674293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.256978 5118 tablet_bootstrap.cc:492] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:42.257789 5118 tablet_bootstrap.cc:654] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.257741 31839 tablet_service.cc:1467] Processing CreateTablet for tablet f787b154dfd44dfe9cf1ef2f996bfffb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.258024 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f787b154dfd44dfe9cf1ef2f996bfffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.259282 5118 tablet_bootstrap.cc:492] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:42.259371 5118 ts_tablet_manager.cc:1367] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:42.259615 29582 tablet_service.cc:1467] Processing CreateTablet for tablet f787b154dfd44dfe9cf1ef2f996bfffb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:42.259863 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f787b154dfd44dfe9cf1ef2f996bfffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.259814 5118 raft_consensus.cc:348] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.259919 5118 raft_consensus.cc:374] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.259953 5118 raft_consensus.cc:729] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.260102 5118 consensus_queue.cc:260] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.260386 5118 ts_tablet_manager.cc:1398] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.261020 5117 tablet_bootstrap.cc:492] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:42.261919 5117 tablet_bootstrap.cc:654] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.263032 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:42.263859 32233 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:42.265167 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:42.272454 29601 raft_consensus.cc:1261] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:42.272961 5135 consensus_queue.cc:1035] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.273177 32120 raft_consensus.cc:1261] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:42.273515 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:42.273980 4831 consensus_queue.cc:1035] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.277516 5184 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:42.278031 4953 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:42.278120 4953 ts_tablet_manager.cc:1367] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:00:42.278589 4953 raft_consensus.cc:348] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.278684 4953 raft_consensus.cc:374] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.278712 4953 raft_consensus.cc:729] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.278815 4953 consensus_queue.cc:260] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.279120 4953 ts_tablet_manager.cc:1398] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.279260 4953 tablet_bootstrap.cc:492] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:42.280175 4953 tablet_bootstrap.cc:654] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.281003 29329 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 4908435d41404a12bc145f39b32a64e0 (table rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5]). Expected version 1 got 0
I20240502 14:00:42.281899 4488 raft_consensus.cc:482] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.281980 4488 raft_consensus.cc:504] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.282224 5185 tablet_bootstrap.cc:492] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:42.282366 4488 leader_election.cc:288] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:42.283285 4977 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:42.283619 4977 tablet.cc:1620] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:42.283845 4976 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:42.283883 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 4908435d41404a12bc145f39b32a64e0 (table rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5]). Expected version 1 got 0
I20240502 14:00:42.284137 4976 tablet.cc:1620] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:42.286829 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1203440714974cb5981e67c2856fb0c7" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:00:42.287271 29537 leader_election.cc:341] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:42.287508 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1203440714974cb5981e67c2856fb0c7" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:42.287694 5185 tablet_bootstrap.cc:654] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
W20240502 14:00:42.287930 29535 leader_election.cc:341] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:42.288005 29535 leader_election.cc:302] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:42.288219 4488 raft_consensus.cc:2726] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:00:42.288995 5117 tablet_bootstrap.cc:492] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:42.289074 5117 ts_tablet_manager.cc:1367] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240502 14:00:42.289582 5117 raft_consensus.cc:348] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.289682 5117 raft_consensus.cc:374] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.289716 5117 raft_consensus.cc:729] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.289827 5117 consensus_queue.cc:260] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.290136 5117 ts_tablet_manager.cc:1398] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.290416 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:42.291222 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5] alter complete (version 1)
I20240502 14:00:42.292757 5184 tablet_bootstrap.cc:654] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.296988 32103 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:42.297668 31839 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:42.311659 4953 tablet_bootstrap.cc:492] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:42.311794 4953 ts_tablet_manager.cc:1367] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240502 14:00:42.312300 4953 raft_consensus.cc:348] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.312425 4953 raft_consensus.cc:374] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.312459 4953 raft_consensus.cc:729] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.312569 4953 consensus_queue.cc:260] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.312836 4953 ts_tablet_manager.cc:1398] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.343492 5187 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:42.343578 5184 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:42.343680 5184 ts_tablet_manager.cc:1367] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.000s
I20240502 14:00:42.343787 5187 ts_tablet_manager.cc:1367] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.090s user 0.000s sys 0.002s
I20240502 14:00:42.344153 5184 raft_consensus.cc:348] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.344295 5184 raft_consensus.cc:374] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.344328 5184 raft_consensus.cc:729] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.344455 5184 consensus_queue.cc:260] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.344527 5187 raft_consensus.cc:348] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.344730 5184 ts_tablet_manager.cc:1398] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.344736 5187 raft_consensus.cc:374] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.344827 5187 raft_consensus.cc:729] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.344996 5187 consensus_queue.cc:260] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.346500 5187 ts_tablet_manager.cc:1398] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:42.348304 5187 tablet_bootstrap.cc:492] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:42.351248 5187 tablet_bootstrap.cc:654] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.352605 5187 tablet_bootstrap.cc:492] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:42.352764 5187 ts_tablet_manager.cc:1367] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:42.353192 5187 raft_consensus.cc:348] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.353355 5187 raft_consensus.cc:374] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.353504 5187 raft_consensus.cc:729] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.353698 5187 consensus_queue.cc:260] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.354013 5187 ts_tablet_manager.cc:1398] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.354321 5187 tablet_bootstrap.cc:492] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:42.355300 5187 tablet_bootstrap.cc:654] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.355618 5185 tablet_bootstrap.cc:492] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:42.355739 5185 ts_tablet_manager.cc:1367] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.000s
I20240502 14:00:42.356235 5185 raft_consensus.cc:348] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.356333 5185 raft_consensus.cc:374] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.356359 5185 raft_consensus.cc:729] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.356487 5185 consensus_queue.cc:260] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.356755 5185 ts_tablet_manager.cc:1398] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:42.358964 29320 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 4908435d41404a12bc145f39b32a64e0 (table rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5]). Expected version 1 got 0
I20240502 14:00:42.361791 5198 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:42.362320 5198 tablet.cc:1620] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:42.371606 5187 tablet_bootstrap.cc:492] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:42.371789 5187 ts_tablet_manager.cc:1367] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:00:42.372246 5187 raft_consensus.cc:348] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.372401 5187 raft_consensus.cc:374] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.372510 5187 raft_consensus.cc:729] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.372725 5187 consensus_queue.cc:260] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.373057 5187 ts_tablet_manager.cc:1398] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.383955 29729 raft_consensus.cc:1203] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:42.386004 5113 raft_consensus.cc:482] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.386152 5113 raft_consensus.cc:504] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.386691 5113 leader_election.cc:288] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:42.387020 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1203440714974cb5981e67c2856fb0c7" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:42.387142 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1203440714974cb5981e67c2856fb0c7" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:42.387379 32255 raft_consensus.cc:2454] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:42.387147 29601 raft_consensus.cc:2454] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:42.387702 31927 leader_election.cc:302] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:42.387924 5113 raft_consensus.cc:2781] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:42.387974 5113 raft_consensus.cc:482] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:42.388018 5113 raft_consensus.cc:3037] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.389145 5113 raft_consensus.cc:504] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.389487 5113 leader_election.cc:288] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:42.390003 32120 consensus_queue.cc:237] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:42.391279 32255 raft_consensus.cc:1261] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:42.391332 29470 raft_consensus.cc:1261] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:42.391371 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1203440714974cb5981e67c2856fb0c7" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:42.391476 29601 raft_consensus.cc:3037] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.391711 3996 consensus_queue.cc:1035] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:42.391821 4789 consensus_queue.cc:1035] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:42.392585 29601 raft_consensus.cc:2454] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:42.391280 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1203440714974cb5981e67c2856fb0c7" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:42.392874 32254 raft_consensus.cc:3037] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.393813 32254 raft_consensus.cc:2454] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:42.394170 31927 leader_election.cc:302] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:42.394461 3996 raft_consensus.cc:2932] T 3a34a1381cf84af6b560101d95447e6d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:42.395732 32254 raft_consensus.cc:2932] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:42.395884 5113 raft_consensus.cc:2781] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:42.395964 5113 raft_consensus.cc:686] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:42.396090 5113 consensus_queue.cc:237] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:42.396531 29470 raft_consensus.cc:2932] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:42.398516 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:42.399690 29324 catalog_manager.cc:5484] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:42.400915 29323 catalog_manager.cc:5484] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:42.400993 5135 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.401095 5135 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.401486 5135 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:42.401783 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787b154dfd44dfe9cf1ef2f996bfffb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:42.401892 32254 raft_consensus.cc:2454] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:42.402122 31796 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:42.402271 5135 raft_consensus.cc:2781] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:42.402315 5135 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:42.402343 5135 raft_consensus.cc:3037] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.403455 5135 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.403509 32254 raft_consensus.cc:1203] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:42.403681 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787b154dfd44dfe9cf1ef2f996bfffb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:42.403797 5135 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:42.403806 29601 raft_consensus.cc:2454] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:42.404104 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787b154dfd44dfe9cf1ef2f996bfffb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:42.404204 32254 raft_consensus.cc:3037] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.404456 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787b154dfd44dfe9cf1ef2f996bfffb" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:42.404553 29601 raft_consensus.cc:3037] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.405145 32254 raft_consensus.cc:2454] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:42.405263 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a34a1381cf84af6b560101d95447e6d with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:42.405581 29601 raft_consensus.cc:2454] T f787b154dfd44dfe9cf1ef2f996bfffb P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:42.405654 31796 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:42.405803 5135 raft_consensus.cc:2781] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:42.405877 5135 raft_consensus.cc:686] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:42.405990 5227 tablet_replica.cc:330] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:42.406013 5135 consensus_queue.cc:237] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:42.406152 5227 raft_consensus.cc:2229] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:42.408830 5227 raft_consensus.cc:2258] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.409286 29320 catalog_manager.cc:5484] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:42.412109 5227 ts_tablet_manager.cc:1875] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:42.412287 4977 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:42.412601 4977 tablet.cc:1620] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:42.412871 4976 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:42.413190 4976 tablet.cc:1620] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:42.414561 5198 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:42.414827 5198 tablet.cc:1620] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:42.419904 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=fe6e63b161c14fdeb7e12d562d283ae5] alter complete (version 2)
I20240502 14:00:42.421612 5227 ts_tablet_manager.cc:1888] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:42.421746 5227 log.cc:1196] T 3a34a1381cf84af6b560101d95447e6d P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/3a34a1381cf84af6b560101d95447e6d
I20240502 14:00:42.422967 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 3a34a1381cf84af6b560101d95447e6d (table rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]) successfully deleted
I20240502 14:00:42.474014 31990 raft_consensus.cc:1203] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:42.481861 4971 raft_consensus.cc:482] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.481999 4971 raft_consensus.cc:504] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.482470 4971 leader_election.cc:288] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:42.483107 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41450335344abe9d261d6a00674293" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:42.483258 32120 raft_consensus.cc:2454] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:42.483214 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41450335344abe9d261d6a00674293" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:42.483376 29729 raft_consensus.cc:2454] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:42.483582 32188 leader_election.cc:302] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:42.483997 4971 raft_consensus.cc:2781] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:42.484063 4971 raft_consensus.cc:482] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:42.484123 4971 raft_consensus.cc:3037] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.485265 4971 raft_consensus.cc:504] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.485633 4971 leader_election.cc:288] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:42.486136 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41450335344abe9d261d6a00674293" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:42.486248 29729 raft_consensus.cc:3037] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.486780 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a41450335344abe9d261d6a00674293" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:42.486918 32120 raft_consensus.cc:3037] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.487313 29729 raft_consensus.cc:2454] T 2a41450335344abe9d261d6a00674293 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:42.487937 32120 raft_consensus.cc:2454] T 2a41450335344abe9d261d6a00674293 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:42.491482 32188 leader_election.cc:302] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:42.493494 4971 raft_consensus.cc:2781] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:42.493595 4971 raft_consensus.cc:686] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:42.493726 4971 consensus_queue.cc:237] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.496171 29320 catalog_manager.cc:5484] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:42.525614 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:42.534195 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:42.534476 4977 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:42.535578 31971 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:42.542886 4976 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:42.547049 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:42.548091 32103 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:42.557688 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=fe6e63b161c14fdeb7e12d562d283ae5] alter complete (version 3)
I20240502 14:00:42.559118 5198 tablet.cc:1722] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:42.559844 32233 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:42.565012 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:42.583034 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:42.588498 31839 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:42.639586 29729 consensus_queue.cc:237] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:42.640763 31859 raft_consensus.cc:1261] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:42.640825 29470 raft_consensus.cc:1261] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:42.641268 4741 consensus_queue.cc:1035] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:42.641440 4722 consensus_queue.cc:1035] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:42.644003 4722 raft_consensus.cc:2932] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:42.644325 29470 raft_consensus.cc:2932] T 1089cd67cbd24551836ccb702ff44d4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:42.646998 31859 raft_consensus.cc:2932] T 1089cd67cbd24551836ccb702ff44d4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:42.647305 29323 catalog_manager.cc:5484] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:42.650007 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 1089cd67cbd24551836ccb702ff44d4a with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:42.651073 5297 tablet_replica.cc:330] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:42.651240 5297 raft_consensus.cc:2229] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:42.651409 5297 raft_consensus.cc:2258] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.652786 5297 ts_tablet_manager.cc:1875] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:42.655158 5297 ts_tablet_manager.cc:1888] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:42.655242 5297 log.cc:1196] T 1089cd67cbd24551836ccb702ff44d4a P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/1089cd67cbd24551836ccb702ff44d4a
I20240502 14:00:42.655864 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 1089cd67cbd24551836ccb702ff44d4a (table rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]) successfully deleted
I20240502 14:00:42.665635 29320 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:42.665778 29320 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:42.667721 29320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 4908435d41404a12bc145f39b32a64e0
I20240502 14:00:42.668357 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet 4908435d41404a12bc145f39b32a64e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:42 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:42.668444 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet 4908435d41404a12bc145f39b32a64e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:42 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:42.668633 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 4908435d41404a12bc145f39b32a64e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:42 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:42.668712 5309 tablet_replica.cc:330] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:42.668871 5309 raft_consensus.cc:2229] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:42.669064 5309 raft_consensus.cc:2258] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.669621 5309 ts_tablet_manager.cc:1875] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:42.671056 5310 tablet_replica.cc:330] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:42.671201 5310 raft_consensus.cc:2229] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:42.671315 5310 raft_consensus.cc:2258] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.671550 5311 tablet_replica.cc:330] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:42.671710 5311 raft_consensus.cc:2229] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:42.671886 5311 raft_consensus.cc:2258] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.672161 5309 ts_tablet_manager.cc:1888] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:42.672242 5309 log.cc:1196] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/4908435d41404a12bc145f39b32a64e0
I20240502 14:00:42.672806 5310 ts_tablet_manager.cc:1875] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:42.672873 5309 ts_tablet_manager.cc:1909] T 4908435d41404a12bc145f39b32a64e0 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:42.674243 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 4908435d41404a12bc145f39b32a64e0 (table rebalancer_dynamic_table_new_name [id=fe6e63b161c14fdeb7e12d562d283ae5]) successfully deleted
I20240502 14:00:42.674911 5310 ts_tablet_manager.cc:1888] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:42.675056 5310 log.cc:1196] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/4908435d41404a12bc145f39b32a64e0
I20240502 14:00:42.675096 5311 ts_tablet_manager.cc:1875] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:42.675422 5310 ts_tablet_manager.cc:1909] T 4908435d41404a12bc145f39b32a64e0 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:42.676021 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 4908435d41404a12bc145f39b32a64e0 (table rebalancer_dynamic_table_new_name [id=fe6e63b161c14fdeb7e12d562d283ae5]) successfully deleted
I20240502 14:00:42.677341 5311 ts_tablet_manager.cc:1888] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:42.677459 5311 log.cc:1196] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/4908435d41404a12bc145f39b32a64e0
I20240502 14:00:42.677804 5311 ts_tablet_manager.cc:1909] T 4908435d41404a12bc145f39b32a64e0 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:42.678601 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 4908435d41404a12bc145f39b32a64e0 (table rebalancer_dynamic_table_new_name [id=fe6e63b161c14fdeb7e12d562d283ae5]) successfully deleted
I20240502 14:00:42.765604 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:42.768440 32233 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:42.770172 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:42.776674 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:42.777168 31971 tablet_service.cc:1467] Processing CreateTablet for tablet be195811fb174147881f53b4b3f70350 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5]), partition=
I20240502 14:00:42.777467 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be195811fb174147881f53b4b3f70350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.777508 31839 tablet_service.cc:1467] Processing CreateTablet for tablet be195811fb174147881f53b4b3f70350 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5]), partition=
I20240502 14:00:42.777742 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be195811fb174147881f53b4b3f70350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.779389 5184 tablet_bootstrap.cc:492] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:42.779562 5117 tablet_bootstrap.cc:492] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:42.780236 5117 tablet_bootstrap.cc:654] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.780237 5184 tablet_bootstrap.cc:654] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.780917 29449 tablet_service.cc:1467] Processing CreateTablet for tablet be195811fb174147881f53b4b3f70350 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5]), partition=
I20240502 14:00:42.781176 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be195811fb174147881f53b4b3f70350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.782953 5184 tablet_bootstrap.cc:492] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:42.783042 5184 ts_tablet_manager.cc:1367] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:42.783535 5184 raft_consensus.cc:348] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.783639 5184 raft_consensus.cc:374] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.783666 5184 raft_consensus.cc:729] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.783775 5184 consensus_queue.cc:260] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.784031 5184 ts_tablet_manager.cc:1398] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.787654 31971 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:42.795068 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:42.797209 5117 tablet_bootstrap.cc:492] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:42.797315 5117 ts_tablet_manager.cc:1367] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:00:42.797812 5117 raft_consensus.cc:348] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.797931 5117 raft_consensus.cc:374] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.797959 5117 raft_consensus.cc:729] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.798066 5117 consensus_queue.cc:260] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.798316 5117 ts_tablet_manager.cc:1398] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:42.799154 5364 tablet_bootstrap.cc:492] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:42.800552 5364 tablet_bootstrap.cc:654] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:42.802026 5364 tablet_bootstrap.cc:492] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:42.802109 5364 ts_tablet_manager.cc:1367] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:42.802531 5364 raft_consensus.cc:348] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.802634 5364 raft_consensus.cc:374] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:42.802668 5364 raft_consensus.cc:729] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.802783 5364 consensus_queue.cc:260] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.803071 5364 ts_tablet_manager.cc:1398] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:42.809007 31839 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:42.812672 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:42.815114 32103 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:42.867655 29729 consensus_queue.cc:237] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:42.868891 32120 raft_consensus.cc:1261] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:42.868981 29601 raft_consensus.cc:1261] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:42.869510 5296 consensus_queue.cc:1035] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:42.869947 4741 consensus_queue.cc:1035] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:42.871790 5296 raft_consensus.cc:2932] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:42.872293 29601 raft_consensus.cc:2932] T 3d2ed5a4845b4bd78b19af6380c77982 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:42.874063 32120 raft_consensus.cc:2932] T 3d2ed5a4845b4bd78b19af6380c77982 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:42.874743 29326 catalog_manager.cc:5484] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:42.877750 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2ed5a4845b4bd78b19af6380c77982 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:42.878041 5297 tablet_replica.cc:330] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:42.878206 5297 raft_consensus.cc:2229] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:42.878382 5297 raft_consensus.cc:2258] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:42.879243 5297 ts_tablet_manager.cc:1875] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:42.881072 5297 ts_tablet_manager.cc:1888] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:42.881125 5297 log.cc:1196] T 3d2ed5a4845b4bd78b19af6380c77982 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d2ed5a4845b4bd78b19af6380c77982
I20240502 14:00:42.882999 4923 raft_consensus.cc:482] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.883127 4923 raft_consensus.cc:504] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.883613 4923 leader_election.cc:288] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:42.883695 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 3d2ed5a4845b4bd78b19af6380c77982 (table rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]) successfully deleted
I20240502 14:00:42.883963 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be195811fb174147881f53b4b3f70350" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:42.884102 31859 raft_consensus.cc:2454] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:42.884478 31925 leader_election.cc:302] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:42.884501 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be195811fb174147881f53b4b3f70350" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:42.884660 4923 raft_consensus.cc:2781] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:42.884654 29470 raft_consensus.cc:2454] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:42.884711 4923 raft_consensus.cc:482] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:42.884752 4923 raft_consensus.cc:3037] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.885769 4923 raft_consensus.cc:504] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.886107 4923 leader_election.cc:288] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:42.886621 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be195811fb174147881f53b4b3f70350" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:42.886741 31859 raft_consensus.cc:3037] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.887825 31859 raft_consensus.cc:2454] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:42.888149 31925 leader_election.cc:302] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:42.888298 4923 raft_consensus.cc:2781] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:42.888378 4923 raft_consensus.cc:686] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:42.888509 4923 consensus_queue.cc:237] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:42.888520 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be195811fb174147881f53b4b3f70350" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:42.888625 29470 raft_consensus.cc:3037] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:42.889664 29470 raft_consensus.cc:2454] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:42.890805 29320 catalog_manager.cc:5484] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:42.911562 5365 consensus_queue.cc:1035] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.915906 5365 consensus_queue.cc:1035] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.919363 5135 consensus_queue.cc:1035] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.925166 4831 consensus_queue.cc:1035] T f787b154dfd44dfe9cf1ef2f996bfffb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.952709 4923 consensus_queue.cc:1035] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.956030 4923 consensus_queue.cc:1035] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:42.999980 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:43.007530 29470 raft_consensus.cc:1261] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:43.007714 31859 raft_consensus.cc:1261] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:43.008235 4923 consensus_queue.cc:1035] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.008518 5113 consensus_queue.cc:1035] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.012784 29328 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet be195811fb174147881f53b4b3f70350 (table rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5]). Expected version 1 got 0
I20240502 14:00:43.013175 29323 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet be195811fb174147881f53b4b3f70350 (table rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5]). Expected version 1 got 0
I20240502 14:00:43.019708 4977 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:43.020057 4977 tablet.cc:1620] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:43.021802 5467 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:43.022063 5468 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:43.022533 5467 tablet.cc:1620] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:43.022732 29322 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet be195811fb174147881f53b4b3f70350 (table rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5]). Expected version 1 got 0
I20240502 14:00:43.022912 5468 tablet.cc:1620] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:43.025985 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5] alter complete (version 1)
I20240502 14:00:43.052585 29329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:43.058979 29713 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:43.063405 32233 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:43.063735 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:43.066239 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 198b0a675f184a268bf714cf778408a4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.066244 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 437ff61fd11046c5ac548d356e5e3402 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.066530 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198b0a675f184a268bf714cf778408a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.066540 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437ff61fd11046c5ac548d356e5e3402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.066479 32103 tablet_service.cc:1467] Processing CreateTablet for tablet abcdb2e98b804235b1196a434113c861 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.066726 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcdb2e98b804235b1196a434113c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.068900 5482 tablet_bootstrap.cc:492] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:43.069941 5482 tablet_bootstrap.cc:654] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.070195 5483 tablet_bootstrap.cc:492] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:43.070413 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 437ff61fd11046c5ac548d356e5e3402 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.070691 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437ff61fd11046c5ac548d356e5e3402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.071435 5483 tablet_bootstrap.cc:654] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.072295 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 198b0a675f184a268bf714cf778408a4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.072526 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198b0a675f184a268bf714cf778408a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.073760 5117 tablet_bootstrap.cc:492] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:43.074908 5117 tablet_bootstrap.cc:654] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.075085 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 437ff61fd11046c5ac548d356e5e3402 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.075367 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437ff61fd11046c5ac548d356e5e3402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.075598 31971 tablet_service.cc:1467] Processing CreateTablet for tablet abcdb2e98b804235b1196a434113c861 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.075820 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcdb2e98b804235b1196a434113c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.076449 29713 tablet_service.cc:1467] Processing CreateTablet for tablet abcdb2e98b804235b1196a434113c861 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.076735 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcdb2e98b804235b1196a434113c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.077746 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 198b0a675f184a268bf714cf778408a4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b27f90a9d23f47febb589aef753349f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.077955 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198b0a675f184a268bf714cf778408a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.078739 5489 tablet_bootstrap.cc:492] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:43.079797 5489 tablet_bootstrap.cc:654] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.079938 5184 tablet_bootstrap.cc:492] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:43.080905 5184 tablet_bootstrap.cc:654] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.083159 31972 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:43.086485 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:43.088441 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:43.088796 5117 tablet_bootstrap.cc:492] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:43.088896 5117 ts_tablet_manager.cc:1367] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:00:43.089370 5117 raft_consensus.cc:348] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.089475 5117 raft_consensus.cc:374] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.089504 5117 raft_consensus.cc:729] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.089618 5117 consensus_queue.cc:260] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.089905 5117 ts_tablet_manager.cc:1398] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.090716 5489 tablet_bootstrap.cc:492] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:43.090837 5489 ts_tablet_manager.cc:1367] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:43.094106 5489 raft_consensus.cc:348] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.094326 5489 raft_consensus.cc:374] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.094404 5489 raft_consensus.cc:729] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.094579 5489 consensus_queue.cc:260] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.094962 5489 ts_tablet_manager.cc:1398] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:00:43.095199 5489 tablet_bootstrap.cc:492] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:43.096462 5489 tablet_bootstrap.cc:654] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.101254 5184 tablet_bootstrap.cc:492] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:43.101354 5184 ts_tablet_manager.cc:1367] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240502 14:00:43.101784 5184 raft_consensus.cc:348] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.101881 5184 raft_consensus.cc:374] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.101909 5184 raft_consensus.cc:729] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.102018 5184 consensus_queue.cc:260] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.102298 5184 ts_tablet_manager.cc:1398] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:43.102636 5184 tablet_bootstrap.cc:492] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:43.103688 5184 tablet_bootstrap.cc:654] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.128022 5184 tablet_bootstrap.cc:492] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:43.128190 5184 ts_tablet_manager.cc:1367] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240502 14:00:43.129019 5184 raft_consensus.cc:348] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.129228 5184 raft_consensus.cc:374] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.129318 5184 raft_consensus.cc:729] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.129547 5184 consensus_queue.cc:260] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.129587 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:43.130008 5184 ts_tablet_manager.cc:1398] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:43.133962 32103 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:43.136948 5467 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:43.137275 5467 tablet.cc:1620] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:43.138208 5468 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:43.138567 5468 tablet.cc:1620] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:43.143945 4977 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:43.144219 4977 tablet.cc:1620] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:43.144426 31839 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:43.148262 5489 tablet_bootstrap.cc:492] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:43.148448 5489 ts_tablet_manager.cc:1367] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.001s
I20240502 14:00:43.148989 5489 raft_consensus.cc:348] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.149253 5489 raft_consensus.cc:374] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.149334 5489 raft_consensus.cc:729] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.149552 5489 consensus_queue.cc:260] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.150010 5489 ts_tablet_manager.cc:1398] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.153409 5483 tablet_bootstrap.cc:492] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:43.153512 5483 ts_tablet_manager.cc:1367] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.000s
I20240502 14:00:43.153975 5483 raft_consensus.cc:348] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.154079 5483 raft_consensus.cc:374] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.154107 5483 raft_consensus.cc:729] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.154235 5483 consensus_queue.cc:260] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.154526 5483 ts_tablet_manager.cc:1398] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.154582 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=36a7db2828c54695b4e476b43f48b2d5] alter complete (version 2)
I20240502 14:00:43.154709 5483 tablet_bootstrap.cc:492] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:43.155153 5481 tablet_bootstrap.cc:492] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:43.155874 5483 tablet_bootstrap.cc:654] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.156410 5481 tablet_bootstrap.cc:654] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.157546 5481 tablet_bootstrap.cc:492] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:43.157632 5481 ts_tablet_manager.cc:1367] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:43.158080 5481 raft_consensus.cc:348] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.158196 5481 raft_consensus.cc:374] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.158097 5482 tablet_bootstrap.cc:492] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:43.158236 5481 raft_consensus.cc:729] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.158367 5481 consensus_queue.cc:260] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.158628 5481 ts_tablet_manager.cc:1398] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.159170 5482 ts_tablet_manager.cc:1367] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.000s
I20240502 14:00:43.159921 5482 raft_consensus.cc:348] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.160059 5482 raft_consensus.cc:374] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.160104 5482 raft_consensus.cc:729] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.160238 5482 consensus_queue.cc:260] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.160549 5482 ts_tablet_manager.cc:1398] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.170588 5483 tablet_bootstrap.cc:492] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:43.170704 5483 ts_tablet_manager.cc:1367] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:00:43.171262 5483 raft_consensus.cc:348] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.171377 5483 raft_consensus.cc:374] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.171406 5483 raft_consensus.cc:729] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.171519 5483 consensus_queue.cc:260] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.171782 5483 ts_tablet_manager.cc:1398] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.175684 5135 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:43.175812 5135 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.176295 5135 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:43.176803 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437ff61fd11046c5ac548d356e5e3402" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:43.176927 29601 raft_consensus.cc:2454] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:43.177197 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437ff61fd11046c5ac548d356e5e3402" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:43.177313 31990 raft_consensus.cc:2454] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:43.177603 31794 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:43.177762 5135 raft_consensus.cc:2781] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:43.177809 5135 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:43.177850 5135 raft_consensus.cc:3037] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.178982 5135 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.179384 5135 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:43.179944 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437ff61fd11046c5ac548d356e5e3402" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:43.179944 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437ff61fd11046c5ac548d356e5e3402" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:43.180064 31990 raft_consensus.cc:3037] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.180064 29601 raft_consensus.cc:3037] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.181059 31990 raft_consensus.cc:2454] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:43.181059 29601 raft_consensus.cc:2454] T 437ff61fd11046c5ac548d356e5e3402 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:43.181829 31794 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:43.181996 5135 raft_consensus.cc:2781] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:43.182089 5135 raft_consensus.cc:686] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:43.182242 5135 consensus_queue.cc:237] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.184806 29329 catalog_manager.cc:5484] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:43.186208 4923 raft_consensus.cc:482] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:43.186303 4923 raft_consensus.cc:504] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.186666 4923 leader_election.cc:288] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:43.187014 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcdb2e98b804235b1196a434113c861" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:43.187163 32122 raft_consensus.cc:2454] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:43.187582 31925 leader_election.cc:302] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:43.187754 4923 raft_consensus.cc:2781] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:43.187798 4923 raft_consensus.cc:482] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:43.187831 4923 raft_consensus.cc:3037] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.188956 4923 raft_consensus.cc:504] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.189301 4923 leader_election.cc:288] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:43.189301 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcdb2e98b804235b1196a434113c861" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:43.189437 29729 raft_consensus.cc:2454] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:43.189742 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcdb2e98b804235b1196a434113c861" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:43.189852 32122 raft_consensus.cc:3037] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.190649 32122 raft_consensus.cc:2454] T abcdb2e98b804235b1196a434113c861 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:43.190943 31925 leader_election.cc:302] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:43.191107 4923 raft_consensus.cc:2781] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:43.191187 4923 raft_consensus.cc:686] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:43.191356 4923 consensus_queue.cc:237] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.192730 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcdb2e98b804235b1196a434113c861" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:43.192847 29729 raft_consensus.cc:3037] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.193696 29329 catalog_manager.cc:5484] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:43.193809 29729 raft_consensus.cc:2454] T abcdb2e98b804235b1196a434113c861 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:43.210778 31990 consensus_queue.cc:237] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.212123 29729 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:00:43.212155 32254 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:00:43.212529 4923 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:43.212755 5466 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:43.214784 5466 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:43.215243 32254 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:43.216622 29729 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:43.219473 29329 catalog_manager.cc:5484] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:43.222155 32103 tablet_service.cc:1514] Processing DeleteTablet for tablet a1116d9f165b488796ab466f9a866471 with delete_type TABLET_DATA_TOMBSTONED (TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:43.223578 5521 tablet_replica.cc:330] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:43.223769 5521 raft_consensus.cc:2229] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:43.223896 5521 raft_consensus.cc:2258] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:43.224335 5521 ts_tablet_manager.cc:1875] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:43.228431 5521 ts_tablet_manager.cc:1888] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:00:43.228519 5521 log.cc:1196] T a1116d9f165b488796ab466f9a866471 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/a1116d9f165b488796ab466f9a866471
I20240502 14:00:43.229113 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet a1116d9f165b488796ab466f9a866471 (table rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]) successfully deleted
I20240502 14:00:43.242091 4789 raft_consensus.cc:482] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:43.242233 4789 raft_consensus.cc:504] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.242765 4789 leader_election.cc:288] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:43.243163 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198b0a675f184a268bf714cf778408a4" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:43.243181 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198b0a675f184a268bf714cf778408a4" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:43.243307 29863 raft_consensus.cc:2454] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:43.243307 29729 raft_consensus.cc:2454] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:43.243595 32057 leader_election.cc:302] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:43.243865 4789 raft_consensus.cc:2781] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:43.243932 4789 raft_consensus.cc:482] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:43.243988 4789 raft_consensus.cc:3037] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.245131 4789 raft_consensus.cc:504] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.245466 4789 leader_election.cc:288] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:43.245875 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198b0a675f184a268bf714cf778408a4" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:43.246014 29863 raft_consensus.cc:3037] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.246019 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198b0a675f184a268bf714cf778408a4" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:43.246158 29729 raft_consensus.cc:3037] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.247025 29863 raft_consensus.cc:2454] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:43.247069 29729 raft_consensus.cc:2454] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:43.247327 32057 leader_election.cc:302] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:43.247493 4789 raft_consensus.cc:2781] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:43.247576 4789 raft_consensus.cc:686] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:43.247723 4789 consensus_queue.cc:237] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.249944 29329 catalog_manager.cc:5484] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:43.257047 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:43.262835 5467 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:43.263450 4977 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:43.263607 5468 tablet.cc:1722] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:43.269447 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=36a7db2828c54695b4e476b43f48b2d5] alter complete (version 3)
I20240502 14:00:43.342124 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:43.349776 31971 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:43.364940 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:43.365244 31839 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:43.366017 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:43.369788 32233 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:43.374007 29329 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:43.374179 29329 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:43.376840 29329 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet be195811fb174147881f53b4b3f70350
I20240502 14:00:43.377529 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet be195811fb174147881f53b4b3f70350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:43 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:43.378059 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet be195811fb174147881f53b4b3f70350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:43 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:43.378281 31971 tablet_service.cc:1514] Processing DeleteTablet for tablet be195811fb174147881f53b4b3f70350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:43 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:43.378597 5572 tablet_replica.cc:330] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:43.380247 32103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:43.382251 5572 raft_consensus.cc:2229] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:43.385598 5572 raft_consensus.cc:2258] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:43.385671 5571 tablet_replica.cc:330] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:43.385862 5571 raft_consensus.cc:2229] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:43.385993 5571 raft_consensus.cc:2258] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:43.386276 5572 ts_tablet_manager.cc:1875] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:43.386170 5570 tablet_replica.cc:330] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:43.386515 5570 raft_consensus.cc:2229] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:43.386745 5570 raft_consensus.cc:2258] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:43.387349 5570 ts_tablet_manager.cc:1875] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:43.390007 5570 ts_tablet_manager.cc:1888] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:43.390115 5570 log.cc:1196] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/be195811fb174147881f53b4b3f70350
I20240502 14:00:43.390184 5572 ts_tablet_manager.cc:1888] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:43.390287 5572 log.cc:1196] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/be195811fb174147881f53b4b3f70350
I20240502 14:00:43.390440 5570 ts_tablet_manager.cc:1909] T be195811fb174147881f53b4b3f70350 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:43.390583 5572 ts_tablet_manager.cc:1909] T be195811fb174147881f53b4b3f70350 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:43.391211 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet be195811fb174147881f53b4b3f70350 (table rebalancer_dynamic_table_new_name [id=36a7db2828c54695b4e476b43f48b2d5]) successfully deleted
I20240502 14:00:43.391973 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet be195811fb174147881f53b4b3f70350 (table rebalancer_dynamic_table_new_name [id=36a7db2828c54695b4e476b43f48b2d5]) successfully deleted
I20240502 14:00:43.394254 29712 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:43.408339 5571 ts_tablet_manager.cc:1875] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:43.417393 5571 ts_tablet_manager.cc:1888] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:43.419525 5571 log.cc:1196] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/be195811fb174147881f53b4b3f70350
I20240502 14:00:43.419983 5571 ts_tablet_manager.cc:1909] T be195811fb174147881f53b4b3f70350 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:43.420765 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet be195811fb174147881f53b4b3f70350 (table rebalancer_dynamic_table_new_name [id=36a7db2828c54695b4e476b43f48b2d5]) successfully deleted
I20240502 14:00:43.450318 29470 consensus_queue.cc:237] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.451695 32254 raft_consensus.cc:1261] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:43.451800 29601 raft_consensus.cc:1261] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:43.452194 5462 consensus_queue.cc:1035] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:43.452390 5461 consensus_queue.cc:1035] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:43.454165 5225 raft_consensus.cc:2932] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.454576 29601 raft_consensus.cc:2932] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.454699 32255 raft_consensus.cc:2932] T a7cc49f0227d491ba7a8df281856b8f8 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.457386 29329 catalog_manager.cc:5484] T a7cc49f0227d491ba7a8df281856b8f8 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.460672 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet a7cc49f0227d491ba7a8df281856b8f8 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:43.461297 5595 tablet_replica.cc:330] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:43.461642 5595 raft_consensus.cc:2229] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:43.461987 5595 raft_consensus.cc:2258] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:43.462517 5595 ts_tablet_manager.cc:1875] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:43.465029 5595 ts_tablet_manager.cc:1888] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:43.465111 5595 log.cc:1196] T a7cc49f0227d491ba7a8df281856b8f8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7cc49f0227d491ba7a8df281856b8f8
I20240502 14:00:43.465622 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet a7cc49f0227d491ba7a8df281856b8f8 (table rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]) successfully deleted
I20240502 14:00:43.478900 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:43.484553 29449 tablet_service.cc:1467] Processing CreateTablet for tablet a95cfae9e71f452eb9f44adf88481034 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8]), partition=
I20240502 14:00:43.484642 29843 tablet_service.cc:1467] Processing CreateTablet for tablet a95cfae9e71f452eb9f44adf88481034 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8]), partition=
I20240502 14:00:43.484858 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95cfae9e71f452eb9f44adf88481034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.484858 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95cfae9e71f452eb9f44adf88481034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.485337 29863 consensus_queue.cc:237] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:43.487103 29601 raft_consensus.cc:1261] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:43.487298 32233 tablet_service.cc:1467] Processing CreateTablet for tablet a95cfae9e71f452eb9f44adf88481034 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8]), partition=
I20240502 14:00:43.487524 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95cfae9e71f452eb9f44adf88481034. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.487740 5480 consensus_queue.cc:1035] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:43.487900 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 925eac6300a44f17bbbf3fe742bc2a26. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:43.489681 5615 tablet_bootstrap.cc:492] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:43.490061 29729 raft_consensus.cc:1261] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:43.490603 5615 tablet_bootstrap.cc:654] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.491010 5481 tablet_bootstrap.cc:492] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:43.490969 5480 consensus_queue.cc:1035] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:43.491876 5481 tablet_bootstrap.cc:654] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.491977 5615 tablet_bootstrap.cc:492] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:43.492075 5615 ts_tablet_manager.cc:1367] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:43.492214 5613 tablet_bootstrap.cc:492] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:43.492537 5615 raft_consensus.cc:348] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.492647 5615 raft_consensus.cc:374] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.492684 5615 raft_consensus.cc:729] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.492816 5615 consensus_queue.cc:260] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.493141 5615 ts_tablet_manager.cc:1398] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:43.494520 5481 tablet_bootstrap.cc:492] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:43.494635 5481 ts_tablet_manager.cc:1367] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:43.495116 5481 raft_consensus.cc:348] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.495234 5481 raft_consensus.cc:374] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.495297 5481 raft_consensus.cc:729] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.495433 5481 consensus_queue.cc:260] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.495734 5481 ts_tablet_manager.cc:1398] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.496465 5613 tablet_bootstrap.cc:654] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.496524 5484 raft_consensus.cc:2932] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:43.497083 29729 raft_consensus.cc:2932] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:43.498891 29601 raft_consensus.cc:2932] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:43.500700 29326 catalog_manager.cc:5484] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:43.515691 5613 tablet_bootstrap.cc:492] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:43.515815 5613 ts_tablet_manager.cc:1367] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:00:43.516342 5613 raft_consensus.cc:348] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.516454 5613 raft_consensus.cc:374] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.516501 5613 raft_consensus.cc:729] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.516630 5613 consensus_queue.cc:260] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.516932 5613 ts_tablet_manager.cc:1398] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.528182 32255 consensus_queue.cc:237] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:43.529990 29470 raft_consensus.cc:1261] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:43.530352 31859 raft_consensus.cc:1261] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:43.531106 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19d48f827a6243448d344ee44a54482d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:43.531224 5365 consensus_queue.cc:1035] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:43.531529 5365 consensus_queue.cc:1035] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:43.534282 4709 raft_consensus.cc:2932] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.534675 29470 raft_consensus.cc:2932] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.536927 31861 raft_consensus.cc:2932] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.539001 29327 catalog_manager.cc:5484] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.564031 29470 consensus_queue.cc:237] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:43.565743 31861 raft_consensus.cc:1261] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:43.565763 32120 raft_consensus.cc:1261] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:43.565780 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b37defc84d34af08f5744bf60a038c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:43.566118 5225 consensus_queue.cc:1035] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:43.566306 5462 consensus_queue.cc:1035] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:43.568413 5225 raft_consensus.cc:2932] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.569025 31861 raft_consensus.cc:2932] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.570474 32120 raft_consensus.cc:2932] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.571395 29327 catalog_manager.cc:5484] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.583490 4923 consensus_queue.cc:1035] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.587867 4923 consensus_queue.cc:1035] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.598073 29601 consensus_queue.cc:237] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.599905 29863 raft_consensus.cc:1261] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:43.600123 29470 raft_consensus.cc:1261] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:43.600361 4655 consensus_queue.cc:1035] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:43.600565 4488 consensus_queue.cc:1035] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:43.601903 29537 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 835183257412496b90defa7ecaa0ff13. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:43.603621 4655 raft_consensus.cc:2932] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.604094 29863 raft_consensus.cc:2932] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.606899 29471 raft_consensus.cc:2932] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.607245 29326 catalog_manager.cc:5484] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.629259 29471 consensus_queue.cc:237] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:43.630982 31990 raft_consensus.cc:1261] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:43.631400 31861 raft_consensus.cc:1261] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:43.631439 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8eb9e9de44a046ffb4594755f44f15b8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:43.632164 5462 consensus_queue.cc:1035] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:43.632196 5217 consensus_queue.cc:1035] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.632406 5461 consensus_queue.cc:1035] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:43.635794 4789 consensus_queue.cc:1035] T 198b0a675f184a268bf714cf778408a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.635871 5462 raft_consensus.cc:2932] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.638092 31990 raft_consensus.cc:2932] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.640877 31861 raft_consensus.cc:2932] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.641400 29329 catalog_manager.cc:5484] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:43.641641 5637 raft_consensus.cc:482] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:43.641739 5637 raft_consensus.cc:504] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.642246 5637 leader_election.cc:288] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:43.642549 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95cfae9e71f452eb9f44adf88481034" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:43.642704 29471 raft_consensus.cc:2454] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:43.642807 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95cfae9e71f452eb9f44adf88481034" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:43.642979 29863 raft_consensus.cc:2454] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:43.643198 32190 leader_election.cc:302] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:43.643419 5365 raft_consensus.cc:2781] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:43.643476 5365 raft_consensus.cc:482] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:43.643525 5365 raft_consensus.cc:3037] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.644685 5365 raft_consensus.cc:504] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.645117 5365 leader_election.cc:288] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:43.645403 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95cfae9e71f452eb9f44adf88481034" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:43.645537 29863 raft_consensus.cc:3037] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.646502 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95cfae9e71f452eb9f44adf88481034" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:43.646589 29863 raft_consensus.cc:2454] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:43.646620 29471 raft_consensus.cc:3037] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.647132 32189 leader_election.cc:302] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:43.647601 5637 raft_consensus.cc:2781] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:43.647689 5637 raft_consensus.cc:686] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:43.647796 29471 raft_consensus.cc:2454] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:43.647812 5637 consensus_queue.cc:237] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:43.651624 29329 catalog_manager.cc:5484] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:43.696696 5135 consensus_queue.cc:1035] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.703891 5135 consensus_queue.cc:1035] T 437ff61fd11046c5ac548d356e5e3402 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.758281 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:43.762578 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:43.767128 29863 raft_consensus.cc:1261] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:43.768059 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:43.768829 29471 raft_consensus.cc:1261] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:43.768990 4709 consensus_queue.cc:1035] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.769457 4709 consensus_queue.cc:1035] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:43.770462 29320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:43.776438 29329 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet a95cfae9e71f452eb9f44adf88481034 (table rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8]). Expected version 1 got 0
I20240502 14:00:43.781198 5767 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:43.781761 5767 tablet.cc:1620] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:43.782234 32103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:43.783922 29329 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet a95cfae9e71f452eb9f44adf88481034 (table rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8]). Expected version 1 got 0
I20240502 14:00:43.786558 31971 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:43.789748 31839 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:43.796236 5768 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:43.796880 5768 tablet.cc:1620] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:43.796890 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8] alter complete (version 1)
I20240502 14:00:43.797618 29329 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet a95cfae9e71f452eb9f44adf88481034 (table rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8]). Expected version 1 got 0
I20240502 14:00:43.798964 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 8579d75d058e463b85fa5c982d74bf3f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.799325 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8579d75d058e463b85fa5c982d74bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.799496 31840 tablet_service.cc:1467] Processing CreateTablet for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.799772 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.801656 5769 tablet_bootstrap.cc:492] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:43.801905 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 962095fe12324c90ae98a69a151438a6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.802189 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962095fe12324c90ae98a69a151438a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.803676 5770 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:43.804726 5770 tablet_bootstrap.cc:654] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.805083 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 962095fe12324c90ae98a69a151438a6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.805352 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962095fe12324c90ae98a69a151438a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.805362 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 8579d75d058e463b85fa5c982d74bf3f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.805613 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8579d75d058e463b85fa5c982d74bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.808023 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 8579d75d058e463b85fa5c982d74bf3f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.808341 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8579d75d058e463b85fa5c982d74bf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.808656 5613 tablet_bootstrap.cc:492] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:43.809784 5613 tablet_bootstrap.cc:654] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.811152 5613 tablet_bootstrap.cc:492] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:43.811172 31972 tablet_service.cc:1467] Processing CreateTablet for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.811244 5613 ts_tablet_manager.cc:1367] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:43.811468 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.811764 5613 raft_consensus.cc:348] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.811875 5613 raft_consensus.cc:374] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.811908 5613 raft_consensus.cc:729] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.812642 29320 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet a95cfae9e71f452eb9f44adf88481034 (table rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8]). Expected version 1 got 0
I20240502 14:00:43.815883 5613 consensus_queue.cc:260] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.819101 5613 ts_tablet_manager.cc:1398] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 14:00:43.820255 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:43.821858 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 962095fe12324c90ae98a69a151438a6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.822145 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962095fe12324c90ae98a69a151438a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.833057 32103 tablet_service.cc:1467] Processing CreateTablet for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:43.833415 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.837679 5769 tablet_bootstrap.cc:654] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.840802 5771 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:43.841221 5772 tablet_bootstrap.cc:492] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:43.842445 5772 tablet_bootstrap.cc:654] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.843489 32233 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:43.843909 5771 tablet.cc:1620] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:43.842921 29712 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:43.861546 5775 tablet_bootstrap.cc:492] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:43.863317 5769 tablet_bootstrap.cc:492] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:43.863464 5769 ts_tablet_manager.cc:1367] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20240502 14:00:43.864132 5769 raft_consensus.cc:348] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.864290 5769 raft_consensus.cc:374] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.864332 5769 raft_consensus.cc:729] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.864471 5769 consensus_queue.cc:260] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.865116 5769 ts_tablet_manager.cc:1398] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:43.865355 5769 tablet_bootstrap.cc:492] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:43.868659 5769 tablet_bootstrap.cc:654] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.870328 5775 tablet_bootstrap.cc:654] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.872071 5772 tablet_bootstrap.cc:492] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:43.872265 5772 ts_tablet_manager.cc:1367] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.002s
I20240502 14:00:43.872895 5772 raft_consensus.cc:348] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.873126 5772 raft_consensus.cc:374] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.873222 5772 raft_consensus.cc:729] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.873447 5772 consensus_queue.cc:260] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.875758 5772 ts_tablet_manager.cc:1398] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:43.876777 5770 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:43.876889 5770 ts_tablet_manager.cc:1367] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.000s
I20240502 14:00:43.877346 5770 raft_consensus.cc:348] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.877467 5770 raft_consensus.cc:374] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.877493 5770 raft_consensus.cc:729] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.877609 5770 consensus_queue.cc:260] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.877890 5770 ts_tablet_manager.cc:1398] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.878052 5770 tablet_bootstrap.cc:492] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:43.879863 5770 tablet_bootstrap.cc:654] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.880393 5769 tablet_bootstrap.cc:492] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:43.880481 5769 ts_tablet_manager.cc:1367] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:00:43.880935 5769 raft_consensus.cc:348] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.881049 5769 raft_consensus.cc:374] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.881075 5769 raft_consensus.cc:729] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.881196 5769 consensus_queue.cc:260] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:43.881467 5769 ts_tablet_manager.cc:1398] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.881618 5769 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:43.882541 5769 tablet_bootstrap.cc:654] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.883700 5769 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:43.883801 5769 ts_tablet_manager.cc:1367] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:43.884380 5775 tablet_bootstrap.cc:492] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:43.884479 5775 ts_tablet_manager.cc:1367] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:00:43.885006 5775 raft_consensus.cc:348] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.885154 5775 raft_consensus.cc:374] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.885215 5775 raft_consensus.cc:729] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.885361 5775 consensus_queue.cc:260] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.885550 5769 raft_consensus.cc:348] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.885648 5769 raft_consensus.cc:374] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.885679 5769 raft_consensus.cc:729] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.885824 5769 consensus_queue.cc:260] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.886268 5769 ts_tablet_manager.cc:1398] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:43.887763 5775 ts_tablet_manager.cc:1398] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:43.888955 5775 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:43.892755 5775 tablet_bootstrap.cc:654] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:43.894037 5775 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:43.894140 5775 ts_tablet_manager.cc:1367] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:00:43.894614 5775 raft_consensus.cc:348] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.894721 5775 raft_consensus.cc:374] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.894747 5775 raft_consensus.cc:729] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.894876 5775 consensus_queue.cc:260] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.895145 5775 ts_tablet_manager.cc:1398] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.901701 5770 tablet_bootstrap.cc:492] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:43.901829 5770 ts_tablet_manager.cc:1367] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.002s
I20240502 14:00:43.902348 5770 raft_consensus.cc:348] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.902449 5770 raft_consensus.cc:374] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:43.902482 5770 raft_consensus.cc:729] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.902616 5770 consensus_queue.cc:260] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.902952 5770 ts_tablet_manager.cc:1398] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:00:43.925738 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:43.931172 5767 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:43.931538 5767 tablet.cc:1620] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:43.931882 5768 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:43.932220 5768 tablet.cc:1620] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:43.933032 5771 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:43.933318 5771 tablet.cc:1620] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:43.939877 5799 ts_tablet_manager.cc:902] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:43.941035 5799 tablet_copy_client.cc:308] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:00:43.943065 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet 19d48f827a6243448d344ee44a54482d from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:39449)
I20240502 14:00:43.943188 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet 19d48f827a6243448d344ee44a54482d from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:39449: session id = d6e2e2c10def4331bec17bd110c9b423-19d48f827a6243448d344ee44a54482d
I20240502 14:00:43.944494 32275 tablet_copy_source_session.cc:205] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:43.944851 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=798827eadd134d22ae5263b466e4cae8] alter complete (version 2)
I20240502 14:00:43.945772 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d48f827a6243448d344ee44a54482d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.948381 5799 tablet_copy_client.cc:783] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:43.948748 5799 tablet_copy_client.cc:647] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:43.950129 5799 tablet_copy_client.cc:523] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:43.951826 5799 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:43.952785 5113 raft_consensus.cc:482] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:43.952896 5113 raft_consensus.cc:504] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.953444 5113 leader_election.cc:288] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:43.953792 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40f25aa6bcc46fca8cd67a6fd4b86b6" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:43.953922 31859 raft_consensus.cc:2454] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:43.953958 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40f25aa6bcc46fca8cd67a6fd4b86b6" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:43.954109 32120 raft_consensus.cc:2454] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:43.954561 31925 leader_election.cc:302] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:43.954761 5113 raft_consensus.cc:2781] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:43.954823 5113 raft_consensus.cc:482] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:43.954913 5113 raft_consensus.cc:3037] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.955847 5113 raft_consensus.cc:504] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.956184 5113 leader_election.cc:288] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:43.956501 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40f25aa6bcc46fca8cd67a6fd4b86b6" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:43.956526 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d40f25aa6bcc46fca8cd67a6fd4b86b6" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:43.956615 32120 raft_consensus.cc:3037] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.956653 31861 raft_consensus.cc:3037] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.957664 32120 raft_consensus.cc:2454] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:43.957676 31861 raft_consensus.cc:2454] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:43.957950 31925 leader_election.cc:302] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:43.958106 5113 raft_consensus.cc:2781] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:43.958197 5113 raft_consensus.cc:686] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:43.958334 5113 consensus_queue.cc:237] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:43.960697 29325 catalog_manager.cc:5484] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:43.963032 5776 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:43.963151 5776 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.963544 5776 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:43.963946 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8579d75d058e463b85fa5c982d74bf3f" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:43.964082 32120 raft_consensus.cc:2454] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:43.964048 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8579d75d058e463b85fa5c982d74bf3f" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:43.964181 31990 raft_consensus.cc:2454] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:43.964382 31794 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:43.964536 5776 raft_consensus.cc:2781] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:43.964584 5776 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:43.964623 5776 raft_consensus.cc:3037] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.965721 5776 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.966063 5776 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:43.966425 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8579d75d058e463b85fa5c982d74bf3f" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:43.966428 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8579d75d058e463b85fa5c982d74bf3f" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:43.966531 31990 raft_consensus.cc:3037] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.966542 32120 raft_consensus.cc:3037] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:43.967129 5799 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:43.967406 5799 tablet_bootstrap.cc:492] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:43.967424 31990 raft_consensus.cc:2454] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:43.967576 32120 raft_consensus.cc:2454] T 8579d75d058e463b85fa5c982d74bf3f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:43.967594 5799 ts_tablet_manager.cc:1367] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20240502 14:00:43.967685 31794 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:43.968046 5776 raft_consensus.cc:2781] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:43.968080 5799 raft_consensus.cc:348] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:43.968163 5776 raft_consensus.cc:686] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:43.968216 5799 raft_consensus.cc:729] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.968310 5776 consensus_queue.cc:237] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:43.968377 5799 consensus_queue.cc:260] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:43.970070 5799 ts_tablet_manager.cc:1398] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:43.970660 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-19d48f827a6243448d344ee44a54482d received from {username='slave'} at 127.22.55.130:39449
I20240502 14:00:43.970746 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-19d48f827a6243448d344ee44a54482d on tablet 19d48f827a6243448d344ee44a54482d with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:43.970677 29325 catalog_manager.cc:5484] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:44.026065 4741 raft_consensus.cc:482] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:44.026208 4741 raft_consensus.cc:504] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:44.026602 4741 leader_election.cc:288] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:44.026935 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962095fe12324c90ae98a69a151438a6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:44.027083 29471 raft_consensus.cc:2454] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:44.027063 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962095fe12324c90ae98a69a151438a6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:44.027232 32120 raft_consensus.cc:2454] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:44.027416 29668 leader_election.cc:302] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:44.027580 4741 raft_consensus.cc:2781] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:44.027639 4741 raft_consensus.cc:482] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:44.027685 4741 raft_consensus.cc:3037] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.028606 4741 raft_consensus.cc:504] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:44.028956 4741 leader_election.cc:288] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:44.029560 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962095fe12324c90ae98a69a151438a6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:44.029660 32120 raft_consensus.cc:3037] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.030781 32120 raft_consensus.cc:2454] T 962095fe12324c90ae98a69a151438a6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:44.031072 29666 leader_election.cc:302] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:44.031432 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962095fe12324c90ae98a69a151438a6" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:44.031567 29471 raft_consensus.cc:3037] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.032511 29471 raft_consensus.cc:2454] T 962095fe12324c90ae98a69a151438a6 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:44.032954 4741 raft_consensus.cc:2781] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:44.033047 4741 raft_consensus.cc:686] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:44.033187 4741 consensus_queue.cc:237] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:44.035435 5799 ts_tablet_manager.cc:902] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.035917 5799 tablet_copy_client.cc:308] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:44.036449 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet 1b37defc84d34af08f5744bf60a038c5 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:46057)
I20240502 14:00:44.036550 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet 1b37defc84d34af08f5744bf60a038c5 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:46057: session id = d6e2e2c10def4331bec17bd110c9b423-1b37defc84d34af08f5744bf60a038c5
I20240502 14:00:44.037567 29491 tablet_copy_source_session.cc:205] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:44.038502 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b37defc84d34af08f5744bf60a038c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.040681 29325 catalog_manager.cc:5484] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:44.040827 5799 tablet_copy_client.cc:783] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:44.040983 5799 tablet_copy_client.cc:647] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:44.042635 5799 tablet_copy_client.cc:523] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:44.044097 5799 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:44.048004 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:44.051115 5799 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:44.051550 5799 tablet_bootstrap.cc:492] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:44.051739 5799 ts_tablet_manager.cc:1367] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240502 14:00:44.052285 5799 raft_consensus.cc:348] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:44.052395 5799 raft_consensus.cc:729] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.052525 5799 consensus_queue.cc:260] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:44.053092 5799 ts_tablet_manager.cc:1398] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:44.053412 5767 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:44.053805 5768 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:44.055717 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-1b37defc84d34af08f5744bf60a038c5 received from {username='slave'} at 127.22.55.130:46057
I20240502 14:00:44.055799 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-1b37defc84d34af08f5744bf60a038c5 on tablet 1b37defc84d34af08f5744bf60a038c5 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:44.058557 29321 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=798827eadd134d22ae5263b466e4cae8] alter complete (version 3)
I20240502 14:00:44.059311 5771 tablet.cc:1722] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:44.076845 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:44.080050 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:44.082489 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:44.086694 32233 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:44.088402 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:44.092167 5852 ts_tablet_manager.cc:902] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:44.092736 5852 tablet_copy_client.cc:308] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:44.093263 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 835183257412496b90defa7ecaa0ff13 from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57283)
I20240502 14:00:44.093377 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 835183257412496b90defa7ecaa0ff13 from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57283: session id = 8973f2527acb48f5ad36589652de73e1-835183257412496b90defa7ecaa0ff13
I20240502 14:00:44.094332 29622 tablet_copy_source_session.cc:205] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:44.095266 5852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835183257412496b90defa7ecaa0ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.097712 5852 tablet_copy_client.cc:783] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:44.097867 5852 tablet_copy_client.cc:647] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:44.103106 31839 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:44.105808 32103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:44.113646 5853 ts_tablet_manager.cc:902] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:44.113799 5799 ts_tablet_manager.cc:902] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.114267 5799 tablet_copy_client.cc:308] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:44.114735 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet 8eb9e9de44a046ffb4594755f44f15b8 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:46057)
I20240502 14:00:44.114879 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet 8eb9e9de44a046ffb4594755f44f15b8 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:46057: session id = d6e2e2c10def4331bec17bd110c9b423-8eb9e9de44a046ffb4594755f44f15b8
I20240502 14:00:44.115808 5852 tablet_copy_client.cc:523] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:44.115984 29491 tablet_copy_source_session.cc:205] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:44.117383 5852 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:44.127296 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9e9de44a046ffb4594755f44f15b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.129779 5799 tablet_copy_client.cc:783] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:44.129976 5799 tablet_copy_client.cc:647] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:44.130983 29712 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:44.146814 5852 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:44.147364 5852 tablet_bootstrap.cc:492] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:44.147627 5852 ts_tablet_manager.cc:1367] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.002s
I20240502 14:00:44.148154 5852 raft_consensus.cc:348] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:44.148358 5852 raft_consensus.cc:729] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.148540 5852 consensus_queue.cc:260] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:44.149672 5852 ts_tablet_manager.cc:1398] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:44.150437 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-835183257412496b90defa7ecaa0ff13 received from {username='slave'} at 127.22.55.136:57283
I20240502 14:00:44.150516 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-835183257412496b90defa7ecaa0ff13 on tablet 835183257412496b90defa7ecaa0ff13 with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:44.152482 5799 tablet_copy_client.cc:523] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:44.156184 5799 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:44.160542 5799 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:44.160813 5799 tablet_bootstrap.cc:492] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:44.160998 5799 ts_tablet_manager.cc:1367] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:00:44.161485 5799 raft_consensus.cc:348] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:44.161645 5799 raft_consensus.cc:729] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.161828 5799 consensus_queue.cc:260] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:44.162603 5799 ts_tablet_manager.cc:1398] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:44.163141 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-8eb9e9de44a046ffb4594755f44f15b8 received from {username='slave'} at 127.22.55.130:46057
I20240502 14:00:44.163226 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-8eb9e9de44a046ffb4594755f44f15b8 on tablet 8eb9e9de44a046ffb4594755f44f15b8 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:44.163275 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:44.163403 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:44.163694 5853 tablet_copy_client.cc:308] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:44.164283 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 925eac6300a44f17bbbf3fe742bc2a26 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:42159)
I20240502 14:00:44.164399 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 925eac6300a44f17bbbf3fe742bc2a26 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:42159: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-925eac6300a44f17bbbf3fe742bc2a26
I20240502 14:00:44.165450 29884 tablet_copy_source_session.cc:205] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:44.165824 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet a95cfae9e71f452eb9f44adf88481034
I20240502 14:00:44.166682 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet a95cfae9e71f452eb9f44adf88481034 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:44 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:44.166975 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet a95cfae9e71f452eb9f44adf88481034 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:44 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:44.167325 29449 tablet_service.cc:1514] Processing DeleteTablet for tablet a95cfae9e71f452eb9f44adf88481034 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:44 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:44.168596 5853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925eac6300a44f17bbbf3fe742bc2a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.171743 5853 tablet_copy_client.cc:783] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:44.173609 5853 tablet_copy_client.cc:647] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:44.174906 5853 tablet_copy_client.cc:523] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:44.177729 5853 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:44.185050 5871 tablet_replica.cc:330] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:44.185205 5869 tablet_replica.cc:330] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:44.185302 5870 tablet_replica.cc:330] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:44.185463 5870 raft_consensus.cc:2229] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:44.185557 5869 raft_consensus.cc:2229] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:44.185696 5870 raft_consensus.cc:2258] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:44.185806 5869 raft_consensus.cc:2258] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:44.186107 5871 raft_consensus.cc:2229] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:44.186257 5871 raft_consensus.cc:2258] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:44.186631 5869 ts_tablet_manager.cc:1875] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:44.187811 5870 ts_tablet_manager.cc:1875] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:44.187109 5871 ts_tablet_manager.cc:1875] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:44.189118 5869 ts_tablet_manager.cc:1888] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:44.189247 5869 log.cc:1196] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/a95cfae9e71f452eb9f44adf88481034
I20240502 14:00:44.189659 5869 ts_tablet_manager.cc:1909] T a95cfae9e71f452eb9f44adf88481034 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:44.190646 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet a95cfae9e71f452eb9f44adf88481034 (table rebalancer_dynamic_table_new_name [id=798827eadd134d22ae5263b466e4cae8]) successfully deleted
I20240502 14:00:44.192010 5870 ts_tablet_manager.cc:1888] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:44.192159 5870 log.cc:1196] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/a95cfae9e71f452eb9f44adf88481034
I20240502 14:00:44.192516 5870 ts_tablet_manager.cc:1909] T a95cfae9e71f452eb9f44adf88481034 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:44.193286 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet a95cfae9e71f452eb9f44adf88481034 (table rebalancer_dynamic_table_new_name [id=798827eadd134d22ae5263b466e4cae8]) successfully deleted
I20240502 14:00:44.194638 5871 ts_tablet_manager.cc:1888] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:44.194715 5871 log.cc:1196] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/a95cfae9e71f452eb9f44adf88481034
I20240502 14:00:44.195230 5871 ts_tablet_manager.cc:1909] T a95cfae9e71f452eb9f44adf88481034 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:44.196163 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet a95cfae9e71f452eb9f44adf88481034 (table rebalancer_dynamic_table_new_name [id=798827eadd134d22ae5263b466e4cae8]) successfully deleted
I20240502 14:00:44.212181 5853 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:44.212502 5853 tablet_bootstrap.cc:492] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:44.212679 5853 ts_tablet_manager.cc:1367] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.000s
I20240502 14:00:44.213096 5853 raft_consensus.cc:348] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:44.213223 5853 raft_consensus.cc:729] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.213389 5853 consensus_queue.cc:260] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:44.213682 5853 ts_tablet_manager.cc:1398] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:44.214165 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-925eac6300a44f17bbbf3fe742bc2a26 received from {username='slave'} at 127.22.55.134:42159
I20240502 14:00:44.214249 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-925eac6300a44f17bbbf3fe742bc2a26 on tablet 925eac6300a44f17bbbf3fe742bc2a26 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:44.268064 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:44.276276 31839 tablet_service.cc:1467] Processing CreateTablet for tablet e7736d44df8444aa9e268de930661813 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece]), partition=
I20240502 14:00:44.276593 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7736d44df8444aa9e268de930661813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.278978 5770 tablet_bootstrap.cc:492] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:44.279203 29712 tablet_service.cc:1467] Processing CreateTablet for tablet e7736d44df8444aa9e268de930661813 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece]), partition=
I20240502 14:00:44.279450 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7736d44df8444aa9e268de930661813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.279893 5770 tablet_bootstrap.cc:654] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.281796 5772 tablet_bootstrap.cc:492] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:44.282590 5770 tablet_bootstrap.cc:492] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:44.282697 5770 ts_tablet_manager.cc:1367] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:44.282744 5772 tablet_bootstrap.cc:654] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.283243 5770 raft_consensus.cc:348] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.283368 5770 raft_consensus.cc:374] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.283401 5770 raft_consensus.cc:729] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.283383 29843 tablet_service.cc:1467] Processing CreateTablet for tablet e7736d44df8444aa9e268de930661813 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece]), partition=
I20240502 14:00:44.283517 5770 consensus_queue.cc:260] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.283620 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7736d44df8444aa9e268de930661813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.283821 5770 ts_tablet_manager.cc:1398] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:44.284821 5772 tablet_bootstrap.cc:492] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:44.284910 5772 ts_tablet_manager.cc:1367] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:44.285357 5772 raft_consensus.cc:348] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.285456 5772 raft_consensus.cc:374] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.285490 5772 raft_consensus.cc:729] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.285604 5772 consensus_queue.cc:260] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.285847 5772 ts_tablet_manager.cc:1398] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:44.292889 5916 tablet_bootstrap.cc:492] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:44.293958 5916 tablet_bootstrap.cc:654] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.295363 5916 tablet_bootstrap.cc:492] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:44.295467 5916 ts_tablet_manager.cc:1367] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:44.297241 5916 raft_consensus.cc:348] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.297379 5916 raft_consensus.cc:374] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.297417 5916 raft_consensus.cc:729] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.297540 5916 consensus_queue.cc:260] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.298928 5916 ts_tablet_manager.cc:1398] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:44.326228 29449 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:44.336472 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:44.342669 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:44.343616 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:44.344214 32103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:44.354665 32233 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:44.363179 29712 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:44.367864 31839 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:44.424472 5776 consensus_queue.cc:1035] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.433548 29600 raft_consensus.cc:1203] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:44.435535 5776 consensus_queue.cc:1035] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.463362 4923 consensus_queue.cc:1035] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.466657 5113 consensus_queue.cc:1035] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.522135 4741 consensus_queue.cc:1035] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.526183 29600 raft_consensus.cc:1203] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:44.537472 4741 consensus_queue.cc:1035] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.587896 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:44.594748 31839 tablet_service.cc:1467] Processing CreateTablet for tablet d000011409c64f1a9fd6b5ad652d9a4a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.595147 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d000011409c64f1a9fd6b5ad652d9a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.597478 5770 tablet_bootstrap.cc:492] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:44.598012 29449 tablet_service.cc:1467] Processing CreateTablet for tablet d000011409c64f1a9fd6b5ad652d9a4a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.598367 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d000011409c64f1a9fd6b5ad652d9a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.598510 5770 tablet_bootstrap.cc:654] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.598500 29582 tablet_service.cc:1467] Processing CreateTablet for tablet d000011409c64f1a9fd6b5ad652d9a4a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.598779 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d000011409c64f1a9fd6b5ad652d9a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.599308 29450 tablet_service.cc:1467] Processing CreateTablet for tablet ddb52469e62d4e6ab5bcc90cea577c9c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.599529 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb52469e62d4e6ab5bcc90cea577c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.601462 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.601747 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.601794 32234 tablet_service.cc:1467] Processing CreateTablet for tablet ddb52469e62d4e6ab5bcc90cea577c9c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.602073 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb52469e62d4e6ab5bcc90cea577c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.604468 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.604761 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.606649 5770 tablet_bootstrap.cc:492] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:44.606743 5770 ts_tablet_manager.cc:1367] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:00:44.606989 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.607257 5770 raft_consensus.cc:348] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.607391 5770 raft_consensus.cc:374] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.607432 5770 raft_consensus.cc:729] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.607551 5770 consensus_queue.cc:260] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.607561 29843 tablet_service.cc:1467] Processing CreateTablet for tablet ddb52469e62d4e6ab5bcc90cea577c9c (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:44.607831 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb52469e62d4e6ab5bcc90cea577c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.607875 5770 ts_tablet_manager.cc:1398] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:44.609270 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:44.609706 5916 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:44.610666 5916 tablet_bootstrap.cc:654] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.612416 5770 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:44.613209 5984 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:44.613263 5770 tablet_bootstrap.cc:654] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.613848 5982 tablet_bootstrap.cc:492] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:44.614962 5982 tablet_bootstrap.cc:654] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.615228 5916 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:44.615319 5916 ts_tablet_manager.cc:1367] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:00:44.615762 5916 raft_consensus.cc:348] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.615877 5916 raft_consensus.cc:374] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.615904 5916 raft_consensus.cc:729] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.616031 5916 consensus_queue.cc:260] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.616261 5916 ts_tablet_manager.cc:1398] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:44.616312 5984 tablet_bootstrap.cc:654] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.617204 5982 tablet_bootstrap.cc:492] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:44.617276 5982 ts_tablet_manager.cc:1367] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:00:44.617827 5983 tablet_bootstrap.cc:492] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:44.619091 5983 tablet_bootstrap.cc:654] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.619246 5982 raft_consensus.cc:348] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.619426 5982 raft_consensus.cc:374] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.619510 5982 raft_consensus.cc:729] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.619715 5982 consensus_queue.cc:260] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.620137 5982 ts_tablet_manager.cc:1398] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:44.621390 5984 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:44.621532 5984 ts_tablet_manager.cc:1367] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:44.622052 5984 raft_consensus.cc:348] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.622282 5984 raft_consensus.cc:374] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.622375 5984 raft_consensus.cc:729] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.622530 5984 consensus_queue.cc:260] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.622867 5984 ts_tablet_manager.cc:1398] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:44.623250 5984 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:44.623435 5983 tablet_bootstrap.cc:492] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:44.623569 5983 ts_tablet_manager.cc:1367] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:44.624079 5983 raft_consensus.cc:348] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.624215 5983 raft_consensus.cc:374] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.624301 5983 raft_consensus.cc:729] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.624464 5984 tablet_bootstrap.cc:654] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.624469 5983 consensus_queue.cc:260] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.624794 5983 ts_tablet_manager.cc:1398] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:44.625012 5983 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:44.626083 5983 tablet_bootstrap.cc:654] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.628633 5776 raft_consensus.cc:482] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:44.628731 5776 raft_consensus.cc:504] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.628902 5770 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:44.628986 5770 ts_tablet_manager.cc:1367] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240502 14:00:44.629120 5776 leader_election.cc:288] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:44.629416 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7736d44df8444aa9e268de930661813" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:44.629469 5770 raft_consensus.cc:348] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.629530 29732 raft_consensus.cc:2454] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:44.629549 5770 raft_consensus.cc:374] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.629570 5770 raft_consensus.cc:729] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.629695 5770 consensus_queue.cc:260] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.629806 31795 leader_election.cc:302] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:44.629949 5776 raft_consensus.cc:2781] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:44.629995 5776 raft_consensus.cc:482] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:44.630018 5770 ts_tablet_manager.cc:1398] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:44.630029 5776 raft_consensus.cc:3037] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.630434 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7736d44df8444aa9e268de930661813" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:44.630542 29862 raft_consensus.cc:2454] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:44.630973 5776 raft_consensus.cc:504] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.631239 5776 leader_election.cc:288] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:44.631561 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7736d44df8444aa9e268de930661813" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:44.631669 29862 raft_consensus.cc:3037] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.632395 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7736d44df8444aa9e268de930661813" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:44.632501 29732 raft_consensus.cc:3037] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.632648 5984 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:44.632740 5984 ts_tablet_manager.cc:1367] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:44.632769 29862 raft_consensus.cc:2454] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:44.633198 5984 raft_consensus.cc:348] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.633450 5984 raft_consensus.cc:374] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.633527 29732 raft_consensus.cc:2454] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:44.633545 5984 raft_consensus.cc:729] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.634023 31795 leader_election.cc:302] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:44.635552 5776 raft_consensus.cc:2781] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:44.635640 5776 raft_consensus.cc:686] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:44.635773 5776 consensus_queue.cc:237] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:44.633680 5984 consensus_queue.cc:260] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.638144 5984 ts_tablet_manager.cc:1398] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:44.641486 5983 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:44.641778 29329 catalog_manager.cc:5484] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:44.641645 5983 ts_tablet_manager.cc:1367] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:44.644302 5983 raft_consensus.cc:348] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.644421 5983 raft_consensus.cc:374] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.644464 5983 raft_consensus.cc:729] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.644588 5983 consensus_queue.cc:260] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.644953 5983 ts_tablet_manager.cc:1398] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:44.645220 5983 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:44.646348 5983 tablet_bootstrap.cc:654] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:44.648360 5983 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:44.648727 5983 ts_tablet_manager.cc:1367] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:44.649914 31990 raft_consensus.cc:1203] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:44.650630 5983 raft_consensus.cc:348] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.650974 5983 raft_consensus.cc:374] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:44.651068 5983 raft_consensus.cc:729] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:44.651283 5983 consensus_queue.cc:260] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.652706 5983 ts_tablet_manager.cc:1398] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:44.655108 32255 raft_consensus.cc:1203] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:44.670111 29600 raft_consensus.cc:1203] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:44.673220 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:44.677052 5680 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:44.677176 5680 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.677683 5680 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.678575 5135 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:44.678678 5135 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.679111 5135 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.679275 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d000011409c64f1a9fd6b5ad652d9a4a" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:44.679390 31859 raft_consensus.cc:2454] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:44.679726 29535 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:44.680068 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d000011409c64f1a9fd6b5ad652d9a4a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:44.680186 29600 raft_consensus.cc:2454] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:44.681321 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d000011409c64f1a9fd6b5ad652d9a4a" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:44.681485 29471 raft_consensus.cc:2454] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:44.684741 31794 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:44.684934 5135 raft_consensus.cc:2781] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:44.684986 5135 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:44.685037 5135 raft_consensus.cc:3037] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.686203 5135 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.686594 5135 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.687135 31839 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:44.688436 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d000011409c64f1a9fd6b5ad652d9a4a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:44.688575 29470 raft_consensus.cc:3037] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.689780 29470 raft_consensus.cc:2454] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:44.690285 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d000011409c64f1a9fd6b5ad652d9a4a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:44.690449 29600 raft_consensus.cc:3037] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.691243 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d000011409c64f1a9fd6b5ad652d9a4a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:44.691360 29471 raft_consensus.cc:2362] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1. Re-sending same reply.
I20240502 14:00:44.691527 29600 raft_consensus.cc:2454] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:44.691615 5680 raft_consensus.cc:2741] T d000011409c64f1a9fd6b5ad652d9a4a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 14:00:44.691962 31796 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:44.692418 5776 raft_consensus.cc:2781] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:44.692440 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:44.692492 5776 raft_consensus.cc:686] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:44.692615 5776 consensus_queue.cc:237] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.694252 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:44.695307 32103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:44.695726 29329 catalog_manager.cc:5484] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:44.700951 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:44.719607 32233 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:44.734194 29712 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:44.788642 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:44.794728 29729 raft_consensus.cc:1261] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:44.795519 29862 raft_consensus.cc:1261] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:44.796582 5776 consensus_queue.cc:1035] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.796873 5776 consensus_queue.cc:1035] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:44.802973 29329 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet e7736d44df8444aa9e268de930661813 (table rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece]). Expected version 1 got 0
I20240502 14:00:44.805440 6011 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:44.805936 6011 tablet.cc:1620] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:44.807439 29327 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet e7736d44df8444aa9e268de930661813 (table rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece]). Expected version 1 got 0
I20240502 14:00:44.809108 29328 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet e7736d44df8444aa9e268de930661813 (table rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece]). Expected version 1 got 0
I20240502 14:00:44.814234 6018 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:44.814720 29328 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet e7736d44df8444aa9e268de930661813 (table rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece]). Expected version 1 got 0
I20240502 14:00:44.814813 6018 tablet.cc:1620] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:44.819002 6012 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:44.819468 5776 raft_consensus.cc:482] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:44.819489 6012 tablet.cc:1620] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:44.819578 5776 raft_consensus.cc:504] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.820034 5776 leader_election.cc:288] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.820974 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece] alter complete (version 1)
I20240502 14:00:44.821476 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afd58ca519e4bc8a6ab00c37abf9f1a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:44.821640 29471 raft_consensus.cc:2454] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:44.824047 31796 leader_election.cc:302] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:44.824391 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afd58ca519e4bc8a6ab00c37abf9f1a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:44.824538 32255 raft_consensus.cc:2454] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:44.825811 5776 raft_consensus.cc:2781] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:44.825886 5776 raft_consensus.cc:482] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:44.825935 5776 raft_consensus.cc:3037] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.827075 5776 raft_consensus.cc:504] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.827492 5776 leader_election.cc:288] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.828985 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afd58ca519e4bc8a6ab00c37abf9f1a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:44.829134 29471 raft_consensus.cc:3037] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.830240 29471 raft_consensus.cc:2454] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:44.830608 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afd58ca519e4bc8a6ab00c37abf9f1a" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:44.830734 32255 raft_consensus.cc:3037] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.831296 31796 leader_election.cc:302] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:44.831776 32255 raft_consensus.cc:2454] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:44.831890 5135 raft_consensus.cc:2781] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:44.831983 5135 raft_consensus.cc:686] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:44.832144 5135 consensus_queue.cc:237] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.835845 29328 catalog_manager.cc:5484] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:44.853219 29862 consensus_queue.cc:237] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:44.854619 31990 raft_consensus.cc:1261] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:44.854738 29600 raft_consensus.cc:1261] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:44.855132 6020 consensus_queue.cc:1035] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:44.855405 6020 consensus_queue.cc:1035] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:44.857537 5620 raft_consensus.cc:2932] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:44.857961 29600 raft_consensus.cc:2932] T 925eac6300a44f17bbbf3fe742bc2a26 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:44.861200 29328 catalog_manager.cc:5484] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:44.864398 31990 raft_consensus.cc:2932] T 925eac6300a44f17bbbf3fe742bc2a26 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:44.865120 5365 raft_consensus.cc:482] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:44.865252 5365 raft_consensus.cc:504] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.865758 5365 leader_election.cc:288] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.866778 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb52469e62d4e6ab5bcc90cea577c9c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:44.866969 29471 raft_consensus.cc:2454] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:44.867740 32190 leader_election.cc:302] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:44.868891 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb52469e62d4e6ab5bcc90cea577c9c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:44.869024 29862 raft_consensus.cc:2454] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:44.869694 5365 raft_consensus.cc:2781] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:44.869762 5365 raft_consensus.cc:482] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:44.869805 5365 raft_consensus.cc:3037] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.870880 5365 raft_consensus.cc:504] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.871258 5365 leader_election.cc:288] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:44.872457 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb52469e62d4e6ab5bcc90cea577c9c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:44.872581 29862 raft_consensus.cc:3037] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.873198 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb52469e62d4e6ab5bcc90cea577c9c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:44.873337 29471 raft_consensus.cc:3037] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:44.873615 29862 raft_consensus.cc:2454] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:44.874311 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet 925eac6300a44f17bbbf3fe742bc2a26 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:44.874572 29471 raft_consensus.cc:2454] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:44.876207 32190 leader_election.cc:302] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:44.876400 5365 raft_consensus.cc:2781] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:44.876783 5365 raft_consensus.cc:686] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:44.876932 5365 consensus_queue.cc:237] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:44.879431 29329 catalog_manager.cc:5484] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:44.883050 6040 tablet_replica.cc:330] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:44.883230 6040 raft_consensus.cc:2229] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:44.883446 6040 raft_consensus.cc:2258] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:44.883962 6040 ts_tablet_manager.cc:1875] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:44.886096 6040 ts_tablet_manager.cc:1888] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:44.894928 6040 log.cc:1196] T 925eac6300a44f17bbbf3fe742bc2a26 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/925eac6300a44f17bbbf3fe742bc2a26
I20240502 14:00:44.896667 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 925eac6300a44f17bbbf3fe742bc2a26 (table rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]) successfully deleted
I20240502 14:00:44.924659 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:44.930742 6011 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:44.931114 6011 tablet.cc:1620] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:44.931517 6012 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:44.931843 6012 tablet.cc:1620] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:44.931807 6018 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:44.932091 6018 tablet.cc:1620] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:44.937783 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=46552d44552e48918f731188766ecece] alter complete (version 2)
I20240502 14:00:45.011605 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:45.016882 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:45.017876 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.023538 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:45.025803 32233 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.038216 29712 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.039142 32103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.044350 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.047060 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:45.062328 6011 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:45.062665 6018 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:45.063243 6012 tablet.cc:1722] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:45.072546 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=46552d44552e48918f731188766ecece] alter complete (version 3)
I20240502 14:00:45.102955 6089 consensus_queue.cc:1035] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.106405 6089 consensus_queue.cc:1035] T d000011409c64f1a9fd6b5ad652d9a4a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.144021 29471 consensus_queue.cc:237] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:45.145423 29862 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:45.145699 31992 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:45.145972 5867 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:45.147111 5867 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:45.148495 5867 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:45.151016 31992 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:45.151181 29862 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:45.153231 29326 catalog_manager.cc:5484] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:45.157895 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1a1a1592e84b3898a06d4360bac846 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:45.158218 6040 tablet_replica.cc:330] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:45.158391 6040 raft_consensus.cc:2229] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.158556 6040 raft_consensus.cc:2258] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.159198 6040 ts_tablet_manager.cc:1875] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:45.161149 6040 ts_tablet_manager.cc:1888] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:45.161214 6040 log.cc:1196] T 6f1a1a1592e84b3898a06d4360bac846 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f1a1a1592e84b3898a06d4360bac846
I20240502 14:00:45.161720 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 6f1a1a1592e84b3898a06d4360bac846 (table rebalance_test_table_7 [id=e5ca7dbc253b44bb873ef49b792c3a10]) successfully deleted
I20240502 14:00:45.175695 29322 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:45.175846 29322 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:45.178542 29322 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet e7736d44df8444aa9e268de930661813
I20240502 14:00:45.179047 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet e7736d44df8444aa9e268de930661813 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:45 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:45.179553 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet e7736d44df8444aa9e268de930661813 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:45 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:45.179556 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet e7736d44df8444aa9e268de930661813 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:45 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:45.179419 6126 tablet_replica.cc:330] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:45.179841 6126 raft_consensus.cc:2229] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:45.179901 6127 tablet_replica.cc:330] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:45.179914 6040 tablet_replica.cc:330] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:45.180044 6040 raft_consensus.cc:2229] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.180053 6126 raft_consensus.cc:2258] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.180044 6127 raft_consensus.cc:2229] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.180162 6040 raft_consensus.cc:2258] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.180354 6127 raft_consensus.cc:2258] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.180883 6040 ts_tablet_manager.cc:1875] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:45.181002 6127 ts_tablet_manager.cc:1875] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:45.181982 6126 ts_tablet_manager.cc:1875] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:45.182837 6040 ts_tablet_manager.cc:1888] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:45.182963 6040 log.cc:1196] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/e7736d44df8444aa9e268de930661813
I20240502 14:00:45.183255 6040 ts_tablet_manager.cc:1909] T e7736d44df8444aa9e268de930661813 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:45.183418 6127 ts_tablet_manager.cc:1888] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:45.183490 6127 log.cc:1196] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/e7736d44df8444aa9e268de930661813
I20240502 14:00:45.183791 6127 ts_tablet_manager.cc:1909] T e7736d44df8444aa9e268de930661813 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:45.184005 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet e7736d44df8444aa9e268de930661813 (table rebalancer_dynamic_table_new_name [id=46552d44552e48918f731188766ecece]) successfully deleted
I20240502 14:00:45.184446 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet e7736d44df8444aa9e268de930661813 (table rebalancer_dynamic_table_new_name [id=46552d44552e48918f731188766ecece]) successfully deleted
I20240502 14:00:45.185433 6126 ts_tablet_manager.cc:1888] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:45.185504 6126 log.cc:1196] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/e7736d44df8444aa9e268de930661813
I20240502 14:00:45.185756 6126 ts_tablet_manager.cc:1909] T e7736d44df8444aa9e268de930661813 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:45.186399 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet e7736d44df8444aa9e268de930661813 (table rebalancer_dynamic_table_new_name [id=46552d44552e48918f731188766ecece]) successfully deleted
I20240502 14:00:45.256433 5985 consensus_queue.cc:1035] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.263316 5365 consensus_queue.cc:1035] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.277258 29712 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.281229 32233 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.281548 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:45.283776 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:45.284408 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:45.288708 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.293255 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:45.300107 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 30fa930173cb4c1298233069b178526d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338]), partition=
I20240502 14:00:45.300185 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 30fa930173cb4c1298233069b178526d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338]), partition=
I20240502 14:00:45.300397 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fa930173cb4c1298233069b178526d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.300419 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fa930173cb4c1298233069b178526d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.303287 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 30fa930173cb4c1298233069b178526d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338]), partition=
I20240502 14:00:45.303550 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fa930173cb4c1298233069b178526d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.308293 6174 tablet_bootstrap.cc:492] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:45.308713 31839 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:45.309402 6174 tablet_bootstrap.cc:654] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.316215 32103 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.341401 6173 tablet_bootstrap.cc:492] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:45.342581 6173 tablet_bootstrap.cc:654] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.347009 6172 tablet_bootstrap.cc:492] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:45.351517 6174 tablet_bootstrap.cc:492] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:45.351639 6174 ts_tablet_manager.cc:1367] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.000s
I20240502 14:00:45.351781 6172 tablet_bootstrap.cc:654] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.352102 6174 raft_consensus.cc:348] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.352223 6174 raft_consensus.cc:374] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.352257 6174 raft_consensus.cc:729] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.352380 6174 consensus_queue.cc:260] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.352666 6174 ts_tablet_manager.cc:1398] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:45.359925 6173 tablet_bootstrap.cc:492] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:45.359947 5776 consensus_queue.cc:1035] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.360071 6173 ts_tablet_manager.cc:1367] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 14:00:45.360605 6173 raft_consensus.cc:348] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.360719 6173 raft_consensus.cc:374] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.360754 6173 raft_consensus.cc:729] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.360891 6173 consensus_queue.cc:260] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.361187 6173 ts_tablet_manager.cc:1398] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:45.365245 5776 consensus_queue.cc:1035] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.365587 6020 raft_consensus.cc:482] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:45.365675 6020 raft_consensus.cc:504] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.366024 6020 leader_election.cc:288] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:45.366412 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa930173cb4c1298233069b178526d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
W20240502 14:00:45.366781 29798 leader_election.cc:341] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:45.370195 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa930173cb4c1298233069b178526d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:45.370373 31990 raft_consensus.cc:2454] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:45.372061 29797 leader_election.cc:302] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:45.372443 6020 raft_consensus.cc:2781] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:45.372510 6020 raft_consensus.cc:482] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:45.372548 6020 raft_consensus.cc:3037] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.373714 6020 raft_consensus.cc:504] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.374078 6020 leader_election.cc:288] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:45.375063 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa930173cb4c1298233069b178526d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:45.375196 31990 raft_consensus.cc:3037] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.376303 31990 raft_consensus.cc:2454] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:45.376334 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa930173cb4c1298233069b178526d" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
W20240502 14:00:45.376778 29798 leader_election.cc:341] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:45.376901 29797 leader_election.cc:302] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:45.382436 6020 raft_consensus.cc:2781] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:45.382551 6020 raft_consensus.cc:686] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:45.382691 6020 consensus_queue.cc:237] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.386611 6172 tablet_bootstrap.cc:492] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:45.389891 29320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:45.390372 6172 ts_tablet_manager.cc:1367] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.001s
I20240502 14:00:45.390935 29322 catalog_manager.cc:5484] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:45.391294 6172 raft_consensus.cc:348] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.393505 6172 raft_consensus.cc:374] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.393586 6172 raft_consensus.cc:729] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.393805 6172 consensus_queue.cc:260] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:45.395130 6172 ts_tablet_manager.cc:1398] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:45.398838 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.399196 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.399161 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.399448 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.399680 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 93e9d6f5e9b84267afa5e019f6bb8c08 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.399719 29713 tablet_service.cc:1467] Processing CreateTablet for tablet b415bd7b51fc4e4a8c6fa6083d989bfe (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.399946 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e9d6f5e9b84267afa5e019f6bb8c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.400555 32102 tablet_service.cc:1467] Processing CreateTablet for tablet 93e9d6f5e9b84267afa5e019f6bb8c08 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.400790 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e9d6f5e9b84267afa5e019f6bb8c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.401808 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.402102 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.402156 29842 tablet_service.cc:1467] Processing CreateTablet for tablet b415bd7b51fc4e4a8c6fa6083d989bfe (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.402381 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b415bd7b51fc4e4a8c6fa6083d989bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.402715 6173 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:45.402887 32101 tablet_service.cc:1467] Processing CreateTablet for tablet b415bd7b51fc4e4a8c6fa6083d989bfe (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.403110 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b415bd7b51fc4e4a8c6fa6083d989bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.403452 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 93e9d6f5e9b84267afa5e019f6bb8c08 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7ba00caa2f624723a352dda66a407d9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:45.403698 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e9d6f5e9b84267afa5e019f6bb8c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.403698 6173 tablet_bootstrap.cc:654] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.404652 6172 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:45.405555 6172 tablet_bootstrap.cc:654] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.405555 6187 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:45.406322 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b415bd7b51fc4e4a8c6fa6083d989bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:45.406517 6187 tablet_bootstrap.cc:654] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.407660 6173 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:45.407742 6173 ts_tablet_manager.cc:1367] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:45.408188 6173 raft_consensus.cc:348] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.408303 6173 raft_consensus.cc:374] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.408334 6173 raft_consensus.cc:729] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.408471 6173 consensus_queue.cc:260] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.408735 6189 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:45.408792 6173 ts_tablet_manager.cc:1398] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:45.408970 6173 tablet_bootstrap.cc:492] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:45.409109 6192 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:45.409776 6189 tablet_bootstrap.cc:654] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.409915 6173 tablet_bootstrap.cc:654] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.410059 6192 tablet_bootstrap.cc:654] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.411159 6192 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:45.411243 6192 ts_tablet_manager.cc:1367] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:45.411710 6172 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:45.411788 6172 ts_tablet_manager.cc:1367] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:45.411711 6192 raft_consensus.cc:348] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.411844 6192 raft_consensus.cc:374] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.411882 6192 raft_consensus.cc:729] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.412042 6189 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:45.412010 6192 consensus_queue.cc:260] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.412215 6172 raft_consensus.cc:348] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.412307 6192 ts_tablet_manager.cc:1398] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:45.412318 6172 raft_consensus.cc:374] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.412348 6172 raft_consensus.cc:729] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.412456 6172 consensus_queue.cc:260] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.412747 6172 ts_tablet_manager.cc:1398] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:45.412914 6172 tablet_bootstrap.cc:492] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:45.412122 6189 ts_tablet_manager.cc:1367] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:45.413834 6172 tablet_bootstrap.cc:654] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.414053 6189 raft_consensus.cc:348] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.414178 6189 raft_consensus.cc:374] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.414209 6189 raft_consensus.cc:729] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.414610 6173 tablet_bootstrap.cc:492] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
W20240502 14:00:45.414687 29911 tablet.cc:2369] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:45.414690 6173 ts_tablet_manager.cc:1367] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:45.414378 6189 consensus_queue.cc:260] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.417775 6173 raft_consensus.cc:348] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.417953 6173 raft_consensus.cc:374] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.418027 6173 raft_consensus.cc:729] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.419715 6189 ts_tablet_manager.cc:1398] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:45.418696 6173 consensus_queue.cc:260] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.421378 6173 ts_tablet_manager.cc:1398] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:45.422379 6172 tablet_bootstrap.cc:492] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:45.422489 6172 ts_tablet_manager.cc:1367] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240502 14:00:45.423274 6172 raft_consensus.cc:348] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.423406 6172 raft_consensus.cc:374] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.423451 6172 raft_consensus.cc:729] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.423560 6172 consensus_queue.cc:260] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.423826 6172 ts_tablet_manager.cc:1398] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:45.427522 6187 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:45.427659 6187 ts_tablet_manager.cc:1367] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20240502 14:00:45.428263 6187 raft_consensus.cc:348] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.428467 6187 raft_consensus.cc:374] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.428551 6187 raft_consensus.cc:729] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.428735 6187 consensus_queue.cc:260] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.429036 6187 ts_tablet_manager.cc:1398] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:45.430300 6187 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:45.431334 6187 tablet_bootstrap.cc:654] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.432557 6187 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:45.432691 6187 ts_tablet_manager.cc:1367] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:45.433041 6187 raft_consensus.cc:348] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.433125 6187 raft_consensus.cc:374] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.433158 6187 raft_consensus.cc:729] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.433284 6187 consensus_queue.cc:260] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.433611 6187 ts_tablet_manager.cc:1398] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:45.433770 6187 tablet_bootstrap.cc:492] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:45.434790 6187 tablet_bootstrap.cc:654] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:45.435804 6187 tablet_bootstrap.cc:492] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:45.435884 6187 ts_tablet_manager.cc:1367] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:45.436264 6187 raft_consensus.cc:348] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.436353 6187 raft_consensus.cc:374] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:45.436376 6187 raft_consensus.cc:729] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:45.436484 6187 consensus_queue.cc:260] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.436795 6187 ts_tablet_manager.cc:1398] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:45.439988 31859 consensus_queue.cc:237] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:45.441123 29732 raft_consensus.cc:1261] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:45.441196 31990 raft_consensus.cc:1261] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:45.441509 6089 consensus_queue.cc:1035] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:45.441681 5135 consensus_queue.cc:1035] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:45.443627 6089 raft_consensus.cc:2932] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:45.444175 5929 raft_consensus.cc:482] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:45.444267 5929 raft_consensus.cc:504] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.444540 31990 raft_consensus.cc:2932] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:45.444705 5929 leader_election.cc:288] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:45.445292 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d6f5e9b84267afa5e019f6bb8c08" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:45.445425 29729 raft_consensus.cc:2454] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:45.445720 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d6f5e9b84267afa5e019f6bb8c08" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:45.445842 32254 raft_consensus.cc:2454] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:45.445961 32057 leader_election.cc:302] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:45.446424 5217 raft_consensus.cc:2781] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:45.446485 5217 raft_consensus.cc:482] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:45.446534 5217 raft_consensus.cc:3037] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.447607 29322 catalog_manager.cc:5484] T df1c6786a80e40a98130a5d9057e2877 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:45.448433 5217 raft_consensus.cc:504] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.448472 29732 raft_consensus.cc:2932] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:45.448798 5217 leader_election.cc:288] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:45.449326 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d6f5e9b84267afa5e019f6bb8c08" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:45.449441 29729 raft_consensus.cc:3037] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.450295 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d6f5e9b84267afa5e019f6bb8c08" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:45.450407 32254 raft_consensus.cc:3037] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.450410 29729 raft_consensus.cc:2454] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:45.451311 32057 leader_election.cc:302] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:45.451416 32254 raft_consensus.cc:2454] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:45.451489 5217 raft_consensus.cc:2781] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:45.451579 5217 raft_consensus.cc:686] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:45.451704 5217 consensus_queue.cc:237] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.453508 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet df1c6786a80e40a98130a5d9057e2877 with delete_type TABLET_DATA_TOMBSTONED (TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:45.453907 6217 tablet_replica.cc:330] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:45.454180 6217 raft_consensus.cc:2229] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.454173 29321 catalog_manager.cc:5484] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:45.454401 6217 raft_consensus.cc:2258] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.454833 6217 ts_tablet_manager.cc:1875] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:45.457072 6217 ts_tablet_manager.cc:1888] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:45.457144 6217 log.cc:1196] T df1c6786a80e40a98130a5d9057e2877 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/df1c6786a80e40a98130a5d9057e2877
I20240502 14:00:45.457684 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet df1c6786a80e40a98130a5d9057e2877 (table rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]) successfully deleted
W20240502 14:00:45.459254 29780 tablet.cc:2369] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:45.464313 5135 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:45.464432 5135 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.464793 5135 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:45.465215 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3062c7a80b284f6c8cb28f8c53c9c0a4" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:45.465215 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3062c7a80b284f6c8cb28f8c53c9c0a4" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:45.465368 32120 raft_consensus.cc:2454] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:45.465368 29863 raft_consensus.cc:2454] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:45.465669 31794 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:45.465833 5135 raft_consensus.cc:2781] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:45.465883 5135 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:45.465919 5135 raft_consensus.cc:3037] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.467120 5135 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.467445 5135 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:45.467793 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3062c7a80b284f6c8cb28f8c53c9c0a4" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:45.467907 32120 raft_consensus.cc:3037] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.468798 32120 raft_consensus.cc:2454] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:45.469226 31794 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:45.469420 5135 raft_consensus.cc:2781] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:45.469516 5135 raft_consensus.cc:686] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:45.469666 5135 consensus_queue.cc:237] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:45.469971 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3062c7a80b284f6c8cb28f8c53c9c0a4" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:45.470129 29863 raft_consensus.cc:3037] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.471191 29863 raft_consensus.cc:2454] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:45.472594 29321 catalog_manager.cc:5484] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:45.516513 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:45.523137 31990 raft_consensus.cc:1261] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:45.523121 29729 raft_consensus.cc:3037] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.524303 29729 raft_consensus.cc:1261] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:45.524688 3671 consensus_queue.cc:1035] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.524827 6020 consensus_queue.cc:1035] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.529433 29325 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet 30fa930173cb4c1298233069b178526d (table rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338]). Expected version 1 got 0
I20240502 14:00:45.530670 29322 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 30fa930173cb4c1298233069b178526d (table rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338]). Expected version 1 got 0
I20240502 14:00:45.532639 6012 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:45.533008 6012 tablet.cc:1620] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:45.533442 6018 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:45.533746 6018 tablet.cc:1620] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:45.534904 29328 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 30fa930173cb4c1298233069b178526d (table rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338]). Expected version 1 got 0
I20240502 14:00:45.539247 6260 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:45.540518 6260 tablet.cc:1620] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:45.543303 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338] alter complete (version 1)
I20240502 14:00:45.645706 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:45.651046 6012 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:45.651394 6012 tablet.cc:1620] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:45.652027 6018 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:45.652230 5217 raft_consensus.cc:482] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:45.652307 6018 tablet.cc:1620] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:45.652323 5217 raft_consensus.cc:504] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.652671 5217 leader_election.cc:288] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:45.653170 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b415bd7b51fc4e4a8c6fa6083d989bfe" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:45.653288 29863 raft_consensus.cc:2454] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:45.654422 6260 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:45.654695 6260 tablet.cc:1620] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:45.655596 32057 leader_election.cc:302] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:45.655815 5217 raft_consensus.cc:2781] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:45.655875 5217 raft_consensus.cc:482] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:45.655926 5217 raft_consensus.cc:3037] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.656949 5217 raft_consensus.cc:504] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.657339 5217 leader_election.cc:288] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:45.657677 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b415bd7b51fc4e4a8c6fa6083d989bfe" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:45.657807 29732 raft_consensus.cc:3037] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.658300 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b415bd7b51fc4e4a8c6fa6083d989bfe" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:45.658823 29732 raft_consensus.cc:2454] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:45.658790 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b415bd7b51fc4e4a8c6fa6083d989bfe" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:45.658934 29863 raft_consensus.cc:3037] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:45.659670 32057 leader_election.cc:302] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:45.659732 29863 raft_consensus.cc:2454] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:45.659878 5217 raft_consensus.cc:2781] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:45.659963 5217 raft_consensus.cc:686] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:45.660097 5217 consensus_queue.cc:237] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:45.661232 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=72ac6f2b26874551abcdc2d048606338] alter complete (version 2)
I20240502 14:00:45.662761 29328 catalog_manager.cc:5484] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:45.691017 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:45.695006 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:45.696667 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.697211 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.697360 32233 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.703318 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.703860 29842 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:45.707348 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:45.764366 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:45.770893 6012 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:45.773043 6018 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:45.776001 6260 tablet.cc:1722] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:45.782429 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=72ac6f2b26874551abcdc2d048606338] alter complete (version 3)
I20240502 14:00:45.803686 31859 consensus_queue.cc:237] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:45.805058 32255 raft_consensus.cc:1261] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:45.805166 29600 raft_consensus.cc:1261] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:45.805473 5135 consensus_queue.cc:1035] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:45.805660 6089 consensus_queue.cc:1035] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:45.807374 6089 raft_consensus.cc:2932] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:45.807732 29600 raft_consensus.cc:2932] T b0fa578318964569b2e10d7f858acd4d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:45.807876 32255 raft_consensus.cc:2932] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:45.810504 29328 catalog_manager.cc:5484] T b0fa578318964569b2e10d7f858acd4d P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:45.813634 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet b0fa578318964569b2e10d7f858acd4d with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:45.815726 6307 tablet_replica.cc:330] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:45.815898 6307 raft_consensus.cc:2229] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.816082 6307 raft_consensus.cc:2258] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.816618 6307 ts_tablet_manager.cc:1875] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:45.818500 6307 ts_tablet_manager.cc:1888] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:45.818571 6307 log.cc:1196] T b0fa578318964569b2e10d7f858acd4d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0fa578318964569b2e10d7f858acd4d
I20240502 14:00:45.819252 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet b0fa578318964569b2e10d7f858acd4d (table rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]) successfully deleted
I20240502 14:00:45.886193 29322 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:45.886371 29322 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:45.888828 29322 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 30fa930173cb4c1298233069b178526d
I20240502 14:00:45.889379 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fa930173cb4c1298233069b178526d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:45 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:45.889703 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fa930173cb4c1298233069b178526d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:45 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:45.889746 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fa930173cb4c1298233069b178526d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:45 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:45.889930 6307 tablet_replica.cc:330] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:45.890061 6365 tablet_replica.cc:330] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:45.890111 6307 raft_consensus.cc:2229] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:45.890197 6365 raft_consensus.cc:2229] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.890311 6365 raft_consensus.cc:2258] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.890309 6307 raft_consensus.cc:2258] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.891347 6307 ts_tablet_manager.cc:1875] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:45.891377 6365 ts_tablet_manager.cc:1875] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:45.893759 6307 ts_tablet_manager.cc:1888] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:45.893823 6307 log.cc:1196] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/30fa930173cb4c1298233069b178526d
I20240502 14:00:45.894075 6307 ts_tablet_manager.cc:1909] T 30fa930173cb4c1298233069b178526d P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:45.894723 6365 ts_tablet_manager.cc:1888] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:45.894896 6365 log.cc:1196] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/30fa930173cb4c1298233069b178526d
I20240502 14:00:45.894922 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 30fa930173cb4c1298233069b178526d (table rebalancer_dynamic_table_new_name [id=72ac6f2b26874551abcdc2d048606338]) successfully deleted
I20240502 14:00:45.895368 6365 ts_tablet_manager.cc:1909] T 30fa930173cb4c1298233069b178526d P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:45.897207 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 30fa930173cb4c1298233069b178526d (table rebalancer_dynamic_table_new_name [id=72ac6f2b26874551abcdc2d048606338]) successfully deleted
I20240502 14:00:45.899055 6364 tablet_replica.cc:330] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:45.899235 6364 raft_consensus.cc:2229] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.899384 6364 raft_consensus.cc:2258] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.900230 6364 ts_tablet_manager.cc:1875] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:45.907444 5217 consensus_queue.cc:1035] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.909425 6364 ts_tablet_manager.cc:1888] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:45.909561 6364 log.cc:1196] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/30fa930173cb4c1298233069b178526d
I20240502 14:00:45.909888 6364 ts_tablet_manager.cc:1909] T 30fa930173cb4c1298233069b178526d P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:45.915001 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 30fa930173cb4c1298233069b178526d (table rebalancer_dynamic_table_new_name [id=72ac6f2b26874551abcdc2d048606338]) successfully deleted
I20240502 14:00:45.919332 5217 consensus_queue.cc:1035] T 93e9d6f5e9b84267afa5e019f6bb8c08 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.962694 29713 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.966061 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:45.966217 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:45.971168 5776 consensus_queue.cc:1035] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.972712 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.978804 5135 consensus_queue.cc:1035] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:45.980726 29842 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:45.987016 32233 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:45.990036 29582 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:45.997089 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:46.000176 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:46.008247 29451 tablet_service.cc:1467] Processing CreateTablet for tablet c38f72337ca04532afc39613f6266a05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c]), partition=
I20240502 14:00:46.008807 31972 tablet_service.cc:1467] Processing CreateTablet for tablet c38f72337ca04532afc39613f6266a05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c]), partition=
I20240502 14:00:46.009055 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38f72337ca04532afc39613f6266a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.009156 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38f72337ca04532afc39613f6266a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.013232 29842 tablet_service.cc:1467] Processing CreateTablet for tablet c38f72337ca04532afc39613f6266a05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c]), partition=
I20240502 14:00:46.013473 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38f72337ca04532afc39613f6266a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.028589 6388 tablet_bootstrap.cc:492] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:46.029767 6388 tablet_bootstrap.cc:654] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.035090 6387 tablet_bootstrap.cc:492] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:46.036561 6387 tablet_bootstrap.cc:654] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.037182 6386 tablet_bootstrap.cc:492] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:46.037861 6388 tablet_bootstrap.cc:492] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:46.038028 6388 ts_tablet_manager.cc:1367] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:46.040108 6388 raft_consensus.cc:348] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.040378 6388 raft_consensus.cc:374] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.040462 6388 raft_consensus.cc:729] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.040678 6388 consensus_queue.cc:260] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.041211 6388 ts_tablet_manager.cc:1398] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:46.042546 6386 tablet_bootstrap.cc:654] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.062079 6386 tablet_bootstrap.cc:492] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:46.062198 6386 ts_tablet_manager.cc:1367] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20240502 14:00:46.062709 6386 raft_consensus.cc:348] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.062837 6386 raft_consensus.cc:374] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.062925 6386 raft_consensus.cc:729] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.063081 6386 consensus_queue.cc:260] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.063400 6386 ts_tablet_manager.cc:1398] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:46.063746 6387 tablet_bootstrap.cc:492] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:46.063843 6387 ts_tablet_manager.cc:1367] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:00:46.064395 6387 raft_consensus.cc:348] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.064611 6387 raft_consensus.cc:374] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.064687 6387 raft_consensus.cc:729] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.064882 6387 consensus_queue.cc:260] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.065248 6387 ts_tablet_manager.cc:1398] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 14:00:46.088294 29518 tablet.cc:2369] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:46.090368 29600 consensus_queue.cc:237] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:46.091596 32255 raft_consensus.cc:1261] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:46.091645 29470 raft_consensus.cc:1261] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:46.092000 6274 consensus_queue.cc:1035] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:46.092190 5680 consensus_queue.cc:1035] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:46.094039 6411 raft_consensus.cc:2932] T 835183257412496b90defa7ecaa0ff13 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.094529 29470 raft_consensus.cc:2932] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.097127 32255 raft_consensus.cc:2932] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.097666 29322 catalog_manager.cc:5484] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.100759 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet 835183257412496b90defa7ecaa0ff13 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:46.101006 6307 tablet_replica.cc:330] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:46.101164 6307 raft_consensus.cc:2229] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:46.101322 6307 raft_consensus.cc:2258] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:46.101926 6307 ts_tablet_manager.cc:1875] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:46.103999 6307 ts_tablet_manager.cc:1888] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:46.104053 6307 log.cc:1196] T 835183257412496b90defa7ecaa0ff13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/835183257412496b90defa7ecaa0ff13
I20240502 14:00:46.104518 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 835183257412496b90defa7ecaa0ff13 (table rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]) successfully deleted
I20240502 14:00:46.142637 29601 consensus_queue.cc:237] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:46.144660 29862 raft_consensus.cc:1261] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:46.145074 5680 consensus_queue.cc:1035] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:46.145527 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd91ed1ae1354a84a0ff668efdbc03d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:46.145562 29729 raft_consensus.cc:1261] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:46.145964 5680 consensus_queue.cc:1035] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:46.148953 6411 raft_consensus.cc:2932] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:46.149556 29729 raft_consensus.cc:2932] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:46.152509 29329 catalog_manager.cc:5484] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:46.155444 29862 raft_consensus.cc:2932] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:46.170545 6368 consensus_queue.cc:1035] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.173928 5217 consensus_queue.cc:1035] T b415bd7b51fc4e4a8c6fa6083d989bfe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.180120 29600 consensus_queue.cc:237] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:46.181514 29470 raft_consensus.cc:1261] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:46.181515 29862 raft_consensus.cc:1261] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:46.181672 29537 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b19374a33aaf41069e98fd035d2b9f00. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:46.181892 6411 consensus_queue.cc:1035] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:46.182060 6274 consensus_queue.cc:1035] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:46.183852 6274 raft_consensus.cc:2932] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.186736 29470 raft_consensus.cc:2932] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.186923 29862 raft_consensus.cc:2932] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.188740 29329 catalog_manager.cc:5484] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:46.192942 6392 raft_consensus.cc:482] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:46.193055 6392 raft_consensus.cc:504] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.193511 6392 leader_election.cc:288] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:46.194203 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38f72337ca04532afc39613f6266a05" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:46.194360 29470 raft_consensus.cc:2454] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:46.194335 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38f72337ca04532afc39613f6266a05" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:46.194475 29862 raft_consensus.cc:2454] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:46.194721 31926 leader_election.cc:302] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:46.195052 6261 raft_consensus.cc:2781] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:46.195111 6261 raft_consensus.cc:482] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:46.195151 6261 raft_consensus.cc:3037] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.196162 6261 raft_consensus.cc:504] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.196525 6261 leader_election.cc:288] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:46.197208 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38f72337ca04532afc39613f6266a05" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:46.197307 29470 raft_consensus.cc:3037] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.197311 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38f72337ca04532afc39613f6266a05" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:46.197415 29862 raft_consensus.cc:3037] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.198235 29470 raft_consensus.cc:2454] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:46.198330 29862 raft_consensus.cc:2454] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:46.198608 31927 leader_election.cc:302] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:46.199043 6261 raft_consensus.cc:2781] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:46.199136 6261 raft_consensus.cc:686] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:46.199268 6261 consensus_queue.cc:237] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.201310 29329 catalog_manager.cc:5484] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:46.221642 29329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:46.228614 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 7ece4fc38b254afeae58936444fe047d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.228705 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 7ece4fc38b254afeae58936444fe047d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.229000 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ece4fc38b254afeae58936444fe047d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.229020 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ece4fc38b254afeae58936444fe047d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.229772 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 90c275c029644bb5be541811dc185f92 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.230093 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c275c029644bb5be541811dc185f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.231127 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2acfdaac334736a065a7850e8293ba (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.231396 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2acfdaac334736a065a7850e8293ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.231714 6473 tablet_bootstrap.cc:492] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:46.232913 6473 tablet_bootstrap.cc:654] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.233317 6474 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:46.234046 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2acfdaac334736a065a7850e8293ba (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.234339 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2acfdaac334736a065a7850e8293ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.234580 6474 tablet_bootstrap.cc:654] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.235774 6473 tablet_bootstrap.cc:492] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:46.235888 6473 ts_tablet_manager.cc:1367] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:46.235986 6474 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:46.236078 6474 ts_tablet_manager.cc:1367] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:46.236372 6473 raft_consensus.cc:348] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.236493 6473 raft_consensus.cc:374] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.236459 6474 raft_consensus.cc:348] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.236526 6473 raft_consensus.cc:729] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.236558 6474 raft_consensus.cc:374] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.236594 6474 raft_consensus.cc:729] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.236624 6388 tablet_bootstrap.cc:492] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:46.236665 6473 consensus_queue.cc:260] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.236735 6474 consensus_queue.cc:260] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.236975 6473 ts_tablet_manager.cc:1398] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:46.237010 6474 ts_tablet_manager.cc:1398] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:46.238240 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 90c275c029644bb5be541811dc185f92 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.238513 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c275c029644bb5be541811dc185f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.240547 6386 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:46.240993 6388 tablet_bootstrap.cc:654] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.241504 6386 tablet_bootstrap.cc:654] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.242285 6388 tablet_bootstrap.cc:492] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:46.242373 6388 ts_tablet_manager.cc:1367] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:46.242862 6388 raft_consensus.cc:348] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.242970 6388 raft_consensus.cc:374] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.243002 6388 raft_consensus.cc:729] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.242990 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 7ece4fc38b254afeae58936444fe047d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.243093 29581 tablet_service.cc:1467] Processing CreateTablet for tablet 90c275c029644bb5be541811dc185f92 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.243136 6388 consensus_queue.cc:260] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.243266 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ece4fc38b254afeae58936444fe047d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.243417 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c275c029644bb5be541811dc185f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.243446 6388 ts_tablet_manager.cc:1398] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:46.243721 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2acfdaac334736a065a7850e8293ba (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:46.243907 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2acfdaac334736a065a7850e8293ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.245993 6475 tablet_bootstrap.cc:492] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:46.247357 6386 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:46.247464 6386 ts_tablet_manager.cc:1367] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:46.247967 6386 raft_consensus.cc:348] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.248112 6386 raft_consensus.cc:374] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.248152 6386 raft_consensus.cc:729] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.248266 6386 consensus_queue.cc:260] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.248451 6475 tablet_bootstrap.cc:654] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.248585 6386 ts_tablet_manager.cc:1398] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:46.248752 6386 tablet_bootstrap.cc:492] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:46.250097 6386 tablet_bootstrap.cc:654] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.250877 6484 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:46.251421 6386 tablet_bootstrap.cc:492] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:46.251518 6386 ts_tablet_manager.cc:1367] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:46.251987 6386 raft_consensus.cc:348] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.252100 6386 raft_consensus.cc:374] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.252132 6386 raft_consensus.cc:729] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.252144 6484 tablet_bootstrap.cc:654] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.252245 6386 consensus_queue.cc:260] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.252565 6386 ts_tablet_manager.cc:1398] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:46.255565 6475 tablet_bootstrap.cc:492] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:46.255697 6475 ts_tablet_manager.cc:1367] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:46.255965 6484 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:46.256047 6484 ts_tablet_manager.cc:1367] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:46.256184 6475 raft_consensus.cc:348] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.256292 6475 raft_consensus.cc:374] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.256323 6475 raft_consensus.cc:729] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.256459 6475 consensus_queue.cc:260] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.256510 6484 raft_consensus.cc:348] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.256608 6484 raft_consensus.cc:374] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.256636 6484 raft_consensus.cc:729] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.256745 6484 consensus_queue.cc:260] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.257022 6484 ts_tablet_manager.cc:1398] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:46.257179 6484 tablet_bootstrap.cc:492] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:46.258263 6484 tablet_bootstrap.cc:654] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.258718 5365 raft_consensus.cc:482] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:46.258808 5365 raft_consensus.cc:504] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.259224 5365 leader_election.cc:288] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:46.259671 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2acfdaac334736a065a7850e8293ba" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:46.259783 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2acfdaac334736a065a7850e8293ba" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:46.259814 29862 raft_consensus.cc:2454] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:46.260159 32189 leader_election.cc:302] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
W20240502 14:00:46.260318 32188 leader_election.cc:341] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:46.260484 5365 raft_consensus.cc:2781] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:46.260533 5365 raft_consensus.cc:482] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:46.260571 5365 raft_consensus.cc:3037] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.261374 6475 ts_tablet_manager.cc:1398] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:46.261579 5365 raft_consensus.cc:504] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.261948 5365 leader_election.cc:288] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:46.262533 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2acfdaac334736a065a7850e8293ba" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:46.262670 6484 tablet_bootstrap.cc:492] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:46.262667 29862 raft_consensus.cc:3037] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.262751 6484 ts_tablet_manager.cc:1367] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:46.263338 6484 raft_consensus.cc:348] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.263438 6484 raft_consensus.cc:374] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.263468 6484 raft_consensus.cc:729] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.263573 6484 consensus_queue.cc:260] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.263806 29862 raft_consensus.cc:2454] T 0d2acfdaac334736a065a7850e8293ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:46.263854 6484 ts_tablet_manager.cc:1398] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:46.264034 6484 tablet_bootstrap.cc:492] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:46.264308 32189 leader_election.cc:302] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:46.264503 5365 raft_consensus.cc:2781] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:46.264585 5365 raft_consensus.cc:686] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:46.264707 5365 consensus_queue.cc:237] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:46.265084 6484 tablet_bootstrap.cc:654] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.265180 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2acfdaac334736a065a7850e8293ba" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:46.265285 29600 raft_consensus.cc:3037] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.266258 29600 raft_consensus.cc:2454] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:46.266891 29322 catalog_manager.cc:5484] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:46.269603 6484 tablet_bootstrap.cc:492] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:46.269697 6484 ts_tablet_manager.cc:1367] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:00:46.270135 6484 raft_consensus.cc:348] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.270213 6484 raft_consensus.cc:374] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.270233 6484 raft_consensus.cc:729] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.270329 6484 consensus_queue.cc:260] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.270588 6484 ts_tablet_manager.cc:1398] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 14:00:46.283500 29649 tablet.cc:2369] T 0d2acfdaac334736a065a7850e8293ba P d6e2e2c10def4331bec17bd110c9b423: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:46.324818 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:46.347895 29862 raft_consensus.cc:1261] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:46.347893 29470 raft_consensus.cc:1261] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:46.348278 6392 consensus_queue.cc:1035] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.348721 6392 consensus_queue.cc:1035] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.351033 29326 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet c38f72337ca04532afc39613f6266a05 (table rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c]). Expected version 1 got 0
I20240502 14:00:46.352344 6523 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:46.352406 6008 mvcc.cc:205] Tried to move back new op lower bound from 7023240996237414400 to 7023240995633704960. Current Snapshot: MvccSnapshot[applied={T|T < 7023240996237414400}]
I20240502 14:00:46.352476 29322 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet c38f72337ca04532afc39613f6266a05 (table rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c]). Expected version 1 got 0
I20240502 14:00:46.352738 6523 tablet.cc:1620] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:46.356042 6524 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:46.356467 6524 tablet.cc:1620] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:46.357070 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c] alter complete (version 1)
I20240502 14:00:46.357290 29326 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet c38f72337ca04532afc39613f6266a05 (table rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c]). Expected version 1 got 0
I20240502 14:00:46.358347 6522 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:46.358619 6522 tablet.cc:1620] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:46.415215 6411 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:46.415344 6411 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.415817 6411 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:46.416211 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece4fc38b254afeae58936444fe047d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:46.416210 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece4fc38b254afeae58936444fe047d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:46.416357 31859 raft_consensus.cc:2454] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:46.416368 31992 raft_consensus.cc:2454] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:46.416674 29535 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:46.416846 6411 raft_consensus.cc:2781] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:46.416905 6411 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:46.416949 6411 raft_consensus.cc:3037] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.417949 6411 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.418306 6411 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:46.418668 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece4fc38b254afeae58936444fe047d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:46.418792 31859 raft_consensus.cc:3037] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.418756 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece4fc38b254afeae58936444fe047d" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:46.418901 31992 raft_consensus.cc:3037] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.419730 31859 raft_consensus.cc:2454] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:46.419880 31992 raft_consensus.cc:2454] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:46.420132 29535 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:46.420406 6411 raft_consensus.cc:2781] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:46.420511 6411 raft_consensus.cc:686] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:46.420650 6411 consensus_queue.cc:237] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:46.423362 29324 catalog_manager.cc:5484] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:46.458988 29324 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:46.464025 6522 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:46.464443 6523 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:46.464715 6522 tablet.cc:1620] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:46.464758 6523 tablet.cc:1620] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:46.464720 6524 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:46.465019 6524 tablet.cc:1620] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:46.468329 6288 raft_consensus.cc:482] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:46.468443 6288 raft_consensus.cc:504] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.468945 6288 leader_election.cc:288] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:46.470477 29324 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=324a26a7cc6b4f719090187e7420b65c] alter complete (version 2)
I20240502 14:00:46.472023 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c275c029644bb5be541811dc185f92" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:46.472151 31992 raft_consensus.cc:2454] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:46.472493 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c275c029644bb5be541811dc185f92" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:46.472627 29600 raft_consensus.cc:2454] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:46.472877 29666 leader_election.cc:302] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:46.473196 6288 raft_consensus.cc:2781] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:46.473268 6288 raft_consensus.cc:482] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:46.473309 6288 raft_consensus.cc:3037] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.474515 6288 raft_consensus.cc:504] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.474838 6288 leader_election.cc:288] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:46.475200 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c275c029644bb5be541811dc185f92" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:46.475200 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c275c029644bb5be541811dc185f92" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:46.475318 31990 raft_consensus.cc:3037] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.475318 29600 raft_consensus.cc:3037] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:46.476276 29600 raft_consensus.cc:2454] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:46.476276 31990 raft_consensus.cc:2454] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:46.476574 29666 leader_election.cc:302] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:46.476738 4741 raft_consensus.cc:2781] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:46.476837 4741 raft_consensus.cc:686] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:46.476960 4741 consensus_queue.cc:237] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:46.479677 29326 catalog_manager.cc:5484] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:46.494868 29582 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:46.502686 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:46.509274 29842 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:46.513450 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:46.530594 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:46.533679 32233 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:46.535125 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:46.550837 29713 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:46.574031 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:46.585006 6522 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:46.585610 6523 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:46.588464 6524 tablet.cc:1722] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:46.592937 29324 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=324a26a7cc6b4f719090187e7420b65c] alter complete (version 3)
I20240502 14:00:46.629555 6555 ts_tablet_manager.cc:902] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:46.630101 6555 tablet_copy_client.cc:308] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:46.630532 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet b19374a33aaf41069e98fd035d2b9f00 from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57283)
I20240502 14:00:46.630633 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet b19374a33aaf41069e98fd035d2b9f00 from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57283: session id = 8973f2527acb48f5ad36589652de73e1-b19374a33aaf41069e98fd035d2b9f00
I20240502 14:00:46.631604 29622 tablet_copy_source_session.cc:205] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:46.632513 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19374a33aaf41069e98fd035d2b9f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.634709 6555 tablet_copy_client.cc:783] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:46.635003 6555 tablet_copy_client.cc:647] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:46.636831 6555 tablet_copy_client.cc:523] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:46.638120 6555 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:46.664324 6555 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:46.664844 6555 tablet_bootstrap.cc:492] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:46.665218 6555 ts_tablet_manager.cc:1367] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.005s
I20240502 14:00:46.665803 6555 raft_consensus.cc:348] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:46.666028 6555 raft_consensus.cc:729] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.666229 6555 consensus_queue.cc:260] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:46.666589 6555 ts_tablet_manager.cc:1398] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:46.667294 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-b19374a33aaf41069e98fd035d2b9f00 received from {username='slave'} at 127.22.55.136:57283
I20240502 14:00:46.667392 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-b19374a33aaf41069e98fd035d2b9f00 on tablet b19374a33aaf41069e98fd035d2b9f00 with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:46.697577 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:46.697851 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:46.701658 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet c38f72337ca04532afc39613f6266a05
I20240502 14:00:46.702489 29842 tablet_service.cc:1514] Processing DeleteTablet for tablet c38f72337ca04532afc39613f6266a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:46 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:46.702829 6593 tablet_replica.cc:330] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:46.703068 6593 raft_consensus.cc:2229] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:46.703096 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet c38f72337ca04532afc39613f6266a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:46 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:46.703261 6593 raft_consensus.cc:2258] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:46.703604 6594 tablet_replica.cc:330] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:46.703917 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet c38f72337ca04532afc39613f6266a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:46 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:46.703786 6594 raft_consensus.cc:2229] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:46.704701 6594 raft_consensus.cc:2258] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:46.705430 6594 ts_tablet_manager.cc:1875] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:46.707986 6593 ts_tablet_manager.cc:1875] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:46.710219 6594 ts_tablet_manager.cc:1888] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:46.710315 6594 log.cc:1196] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/c38f72337ca04532afc39613f6266a05
I20240502 14:00:46.710343 6593 ts_tablet_manager.cc:1888] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:46.710423 6593 log.cc:1196] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/c38f72337ca04532afc39613f6266a05
I20240502 14:00:46.710711 6593 ts_tablet_manager.cc:1909] T c38f72337ca04532afc39613f6266a05 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:46.710707 6594 ts_tablet_manager.cc:1909] T c38f72337ca04532afc39613f6266a05 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:46.710953 6595 tablet_replica.cc:330] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:46.711158 6595 raft_consensus.cc:2229] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:46.711414 6595 raft_consensus.cc:2258] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:46.711735 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet c38f72337ca04532afc39613f6266a05 (table rebalancer_dynamic_table_new_name [id=324a26a7cc6b4f719090187e7420b65c]) successfully deleted
I20240502 14:00:46.712148 6595 ts_tablet_manager.cc:1875] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:46.714264 6595 ts_tablet_manager.cc:1888] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:46.714334 6595 log.cc:1196] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/c38f72337ca04532afc39613f6266a05
I20240502 14:00:46.714608 6595 ts_tablet_manager.cc:1909] T c38f72337ca04532afc39613f6266a05 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:46.715306 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet c38f72337ca04532afc39613f6266a05 (table rebalancer_dynamic_table_new_name [id=324a26a7cc6b4f719090187e7420b65c]) successfully deleted
I20240502 14:00:46.715507 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet c38f72337ca04532afc39613f6266a05 (table rebalancer_dynamic_table_new_name [id=324a26a7cc6b4f719090187e7420b65c]) successfully deleted
I20240502 14:00:46.727921 6592 ts_tablet_manager.cc:902] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:46.728747 6592 tablet_copy_client.cc:308] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:46.729621 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet bd91ed1ae1354a84a0ff668efdbc03d9 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37007)
I20240502 14:00:46.729739 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet bd91ed1ae1354a84a0ff668efdbc03d9 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37007: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-bd91ed1ae1354a84a0ff668efdbc03d9
I20240502 14:00:46.730813 29622 tablet_copy_source_session.cc:205] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:46.733130 6592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd91ed1ae1354a84a0ff668efdbc03d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.735999 6592 tablet_copy_client.cc:783] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:46.736163 6592 tablet_copy_client.cc:647] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:46.740916 6592 tablet_copy_client.cc:523] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:46.742779 29842 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:46.743141 6592 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:46.745549 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:46.747467 5365 consensus_queue.cc:1035] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.760746 32233 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:46.765295 29582 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:46.769136 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:46.791779 29713 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:46.794785 5365 consensus_queue.cc:1035] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.797541 6592 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:46.798173 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:46.798552 6592 tablet_bootstrap.cc:492] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:46.798796 6592 ts_tablet_manager.cc:1367] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.056s user 0.028s sys 0.001s
I20240502 14:00:46.799327 6592 raft_consensus.cc:348] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:46.799518 6592 raft_consensus.cc:729] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.800573 6592 consensus_queue.cc:260] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:46.801910 6592 ts_tablet_manager.cc:1398] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:46.802443 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-bd91ed1ae1354a84a0ff668efdbc03d9 received from {username='slave'} at 127.22.55.135:37007
I20240502 14:00:46.802532 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-bd91ed1ae1354a84a0ff668efdbc03d9 on tablet bd91ed1ae1354a84a0ff668efdbc03d9 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:46.803704 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:46.811934 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 9076cc42e8694807a0af8b0851b17935 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]), partition=
I20240502 14:00:46.812247 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9076cc42e8694807a0af8b0851b17935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.812875 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 9076cc42e8694807a0af8b0851b17935 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]), partition=
I20240502 14:00:46.813141 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9076cc42e8694807a0af8b0851b17935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.815194 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 9076cc42e8694807a0af8b0851b17935 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]), partition=
I20240502 14:00:46.815290 6621 tablet_bootstrap.cc:492] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:46.815446 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9076cc42e8694807a0af8b0851b17935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:46.816290 6621 tablet_bootstrap.cc:654] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.817723 6623 tablet_bootstrap.cc:492] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:46.818639 6623 tablet_bootstrap.cc:654] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.822443 6623 tablet_bootstrap.cc:492] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:46.822621 6623 ts_tablet_manager.cc:1367] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:00:46.823594 6623 raft_consensus.cc:348] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:46.823997 6623 raft_consensus.cc:374] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.824115 6623 raft_consensus.cc:729] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.824335 6623 consensus_queue.cc:260] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:46.825371 6623 ts_tablet_manager.cc:1398] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:46.826308 6620 tablet_bootstrap.cc:492] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:46.827690 6620 tablet_bootstrap.cc:654] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:46.833834 6621 tablet_bootstrap.cc:492] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:46.833936 6621 ts_tablet_manager.cc:1367] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:00:46.834378 6621 raft_consensus.cc:348] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:46.834508 6621 raft_consensus.cc:374] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.834543 6621 raft_consensus.cc:729] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.834656 6621 consensus_queue.cc:260] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:46.834959 6621 ts_tablet_manager.cc:1398] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:46.836625 6620 tablet_bootstrap.cc:492] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:46.837250 6620 ts_tablet_manager.cc:1367] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240502 14:00:46.837817 6620 raft_consensus.cc:348] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:46.837992 6620 raft_consensus.cc:374] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:46.838104 6620 raft_consensus.cc:729] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:46.838301 6620 consensus_queue.cc:260] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:46.838621 6620 ts_tablet_manager.cc:1398] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:46.843652 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:46.873988 4741 consensus_queue.cc:1035] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.877632 6288 consensus_queue.cc:1035] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.880617 6411 consensus_queue.cc:1035] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:46.883080 6411 consensus_queue.cc:1035] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.014662 6613 raft_consensus.cc:482] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:47.014791 6613 raft_consensus.cc:504] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.015336 6613 leader_election.cc:288] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:47.015759 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9076cc42e8694807a0af8b0851b17935" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:47.015827 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9076cc42e8694807a0af8b0851b17935" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:47.015914 29862 raft_consensus.cc:2454] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:47.015944 29732 raft_consensus.cc:2454] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:47.016237 31795 leader_election.cc:302] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:47.016422 6615 raft_consensus.cc:2781] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:47.016487 6615 raft_consensus.cc:482] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:47.016536 6615 raft_consensus.cc:3037] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.017580 6615 raft_consensus.cc:504] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.017917 6615 leader_election.cc:288] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:47.018208 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9076cc42e8694807a0af8b0851b17935" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:47.018340 29862 raft_consensus.cc:3037] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.018461 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9076cc42e8694807a0af8b0851b17935" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:47.018574 29732 raft_consensus.cc:3037] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.019243 29862 raft_consensus.cc:2454] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:47.019447 29732 raft_consensus.cc:2454] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:47.019526 31795 leader_election.cc:302] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:47.019691 6613 raft_consensus.cc:2781] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:47.019814 6613 raft_consensus.cc:686] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:47.019959 6613 consensus_queue.cc:237] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.022425 29323 catalog_manager.cc:5484] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:47.038254 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:47.043071 31972 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:47.048131 32254 raft_consensus.cc:1203] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:47.049798 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:47.050822 29713 tablet_service.cc:1467] Processing CreateTablet for tablet e78dc2ec29ad408e9665da00778743e2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.050838 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 8151ac615c84417cab1b210a95a0f32c (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.051177 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8151ac615c84417cab1b210a95a0f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.051177 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78dc2ec29ad408e9665da00778743e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.051724 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 70e59a9a97d9411e963d470b235706bd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.051973 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e59a9a97d9411e963d470b235706bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.053751 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 8151ac615c84417cab1b210a95a0f32c (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.054024 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8151ac615c84417cab1b210a95a0f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.056000 6623 tablet_bootstrap.cc:492] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:47.056165 29843 tablet_service.cc:1467] Processing CreateTablet for tablet e78dc2ec29ad408e9665da00778743e2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.056386 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78dc2ec29ad408e9665da00778743e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.056967 6623 tablet_bootstrap.cc:654] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.058676 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 8151ac615c84417cab1b210a95a0f32c (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.058969 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8151ac615c84417cab1b210a95a0f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.059175 29843 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:47.059151 31839 tablet_service.cc:1467] Processing CreateTablet for tablet e78dc2ec29ad408e9665da00778743e2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.059433 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78dc2ec29ad408e9665da00778743e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.061283 6690 tablet_bootstrap.cc:492] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:47.061354 6620 tablet_bootstrap.cc:492] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:47.062315 6620 tablet_bootstrap.cc:654] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.063979 6621 tablet_bootstrap.cc:492] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:47.064131 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 70e59a9a97d9411e963d470b235706bd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.064436 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e59a9a97d9411e963d470b235706bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.064491 6690 tablet_bootstrap.cc:654] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.065315 6621 tablet_bootstrap.cc:654] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.066807 6690 tablet_bootstrap.cc:492] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:47.066972 6690 ts_tablet_manager.cc:1367] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:47.067504 6690 raft_consensus.cc:348] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.067631 6690 raft_consensus.cc:374] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.067670 6690 raft_consensus.cc:729] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.067799 6690 consensus_queue.cc:260] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.068096 6690 ts_tablet_manager.cc:1398] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:47.069458 29713 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:47.072463 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 70e59a9a97d9411e963d470b235706bd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=71a8d6a658c345389465f4d5cfb1af6f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.072782 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70e59a9a97d9411e963d470b235706bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.074455 6694 tablet_bootstrap.cc:492] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:47.075762 6620 tablet_bootstrap.cc:492] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:47.075860 6620 ts_tablet_manager.cc:1367] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240502 14:00:47.076330 6620 raft_consensus.cc:348] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.076462 6620 raft_consensus.cc:374] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.076490 6620 raft_consensus.cc:729] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.076596 6620 consensus_queue.cc:260] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.076861 6620 ts_tablet_manager.cc:1398] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:47.082149 6621 tablet_bootstrap.cc:492] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:47.082275 6621 ts_tablet_manager.cc:1367] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 14:00:47.082834 6621 raft_consensus.cc:348] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.083036 6621 raft_consensus.cc:374] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.083076 6621 raft_consensus.cc:729] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.083202 6621 consensus_queue.cc:260] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.085011 6621 ts_tablet_manager.cc:1398] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:47.085721 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:47.087464 6694 tablet_bootstrap.cc:654] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.089715 6623 tablet_bootstrap.cc:492] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:47.089821 6623 ts_tablet_manager.cc:1367] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.002s
I20240502 14:00:47.090348 6623 raft_consensus.cc:348] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.090467 6623 raft_consensus.cc:374] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.090492 6623 raft_consensus.cc:729] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.090603 6623 consensus_queue.cc:260] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.090905 6623 ts_tablet_manager.cc:1398] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:47.091099 6623 tablet_bootstrap.cc:492] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:47.092088 6623 tablet_bootstrap.cc:654] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.098062 6694 tablet_bootstrap.cc:492] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:47.098249 6694 ts_tablet_manager.cc:1367] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 14:00:47.099028 6694 raft_consensus.cc:348] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.099238 6694 raft_consensus.cc:374] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.099320 6694 raft_consensus.cc:729] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.099519 6694 consensus_queue.cc:260] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.099961 6694 ts_tablet_manager.cc:1398] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:47.107549 6687 tablet_bootstrap.cc:492] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:47.108762 6687 tablet_bootstrap.cc:654] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.109871 6623 tablet_bootstrap.cc:492] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:47.110037 6623 ts_tablet_manager.cc:1367] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:00:47.110597 6623 raft_consensus.cc:348] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.111119 6623 raft_consensus.cc:374] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.111208 6623 raft_consensus.cc:729] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.111560 29582 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:47.111392 6623 consensus_queue.cc:260] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.112903 6623 ts_tablet_manager.cc:1398] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:47.114471 32233 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:47.115938 31839 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:47.122645 6688 tablet_bootstrap.cc:492] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:47.125728 6688 tablet_bootstrap.cc:654] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.131189 6687 tablet_bootstrap.cc:492] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:47.131363 6687 ts_tablet_manager.cc:1367] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20240502 14:00:47.131852 6687 raft_consensus.cc:348] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.132056 6687 raft_consensus.cc:374] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.132136 6687 raft_consensus.cc:729] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.132309 6687 consensus_queue.cc:260] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.132658 6687 ts_tablet_manager.cc:1398] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:47.132851 6687 tablet_bootstrap.cc:492] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:47.134160 6687 tablet_bootstrap.cc:654] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.143229 6687 tablet_bootstrap.cc:492] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:47.143433 6687 ts_tablet_manager.cc:1367] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240502 14:00:47.143954 6687 raft_consensus.cc:348] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.144150 6687 raft_consensus.cc:374] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.144227 6687 raft_consensus.cc:729] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.144421 6687 consensus_queue.cc:260] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.144840 6687 ts_tablet_manager.cc:1398] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:47.148958 5620 raft_consensus.cc:482] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:47.149091 5620 raft_consensus.cc:504] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.149592 5620 leader_election.cc:288] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:47.150066 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78dc2ec29ad408e9665da00778743e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:47.150213 29729 raft_consensus.cc:2454] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:47.150974 29798 leader_election.cc:302] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:47.151584 5620 raft_consensus.cc:2781] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:47.151644 5620 raft_consensus.cc:482] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:47.151679 5620 raft_consensus.cc:3037] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.152042 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78dc2ec29ad408e9665da00778743e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:47.152168 31859 raft_consensus.cc:2454] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:47.152793 5620 raft_consensus.cc:504] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.153131 5620 leader_election.cc:288] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:47.154716 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78dc2ec29ad408e9665da00778743e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:47.154922 6688 tablet_bootstrap.cc:492] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:47.155038 6688 ts_tablet_manager.cc:1367] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240502 14:00:47.155511 29729 raft_consensus.cc:3037] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.155611 6688 raft_consensus.cc:348] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.155719 6688 raft_consensus.cc:374] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.155751 6688 raft_consensus.cc:729] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.155884 6688 consensus_queue.cc:260] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.156203 6688 ts_tablet_manager.cc:1398] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:47.156550 29729 raft_consensus.cc:2454] T e78dc2ec29ad408e9665da00778743e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:47.157037 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78dc2ec29ad408e9665da00778743e2" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:47.157155 31859 raft_consensus.cc:3037] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.158200 31859 raft_consensus.cc:2454] T e78dc2ec29ad408e9665da00778743e2 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:47.158887 29797 leader_election.cc:302] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:47.160424 5620 raft_consensus.cc:2781] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:47.160519 5620 raft_consensus.cc:686] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:47.160653 5620 consensus_queue.cc:237] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:47.163635 29323 catalog_manager.cc:5484] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:47.170506 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:47.176230 29729 raft_consensus.cc:1261] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:47.176793 6613 consensus_queue.cc:1035] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.177349 32122 raft_consensus.cc:1203] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:47.177592 29862 raft_consensus.cc:1261] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:47.178404 6613 consensus_queue.cc:1035] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.181298 29327 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet 9076cc42e8694807a0af8b0851b17935 (table rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]). Expected version 1 got 0
I20240502 14:00:47.183640 29327 catalog_manager.cc:5576] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105) does not have the latest schema for tablet 9076cc42e8694807a0af8b0851b17935 (table rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]). Expected version 1 got 0
I20240502 14:00:47.183665 6707 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:47.184120 6707 tablet.cc:1620] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:47.185753 6708 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:47.186350 6708 tablet.cc:1620] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:47.188263 29323 catalog_manager.cc:5576] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915) does not have the latest schema for tablet 9076cc42e8694807a0af8b0851b17935 (table rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]). Expected version 1 got 0
I20240502 14:00:47.188727 6710 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:47.189010 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae] alter complete (version 1)
I20240502 14:00:47.189678 6710 tablet.cc:1620] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:47.194449 6618 raft_consensus.cc:482] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:47.194564 6618 raft_consensus.cc:504] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.195106 6618 leader_election.cc:288] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:47.195518 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8151ac615c84417cab1b210a95a0f32c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:47.195665 29862 raft_consensus.cc:2454] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:47.195955 32189 leader_election.cc:302] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:47.196107 6618 raft_consensus.cc:2781] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:47.196166 6618 raft_consensus.cc:482] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:47.196230 6618 raft_consensus.cc:3037] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.196292 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8151ac615c84417cab1b210a95a0f32c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:47.196431 32122 raft_consensus.cc:2454] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:47.197242 6618 raft_consensus.cc:504] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.197566 6618 leader_election.cc:288] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:47.197827 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8151ac615c84417cab1b210a95a0f32c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:47.197930 29862 raft_consensus.cc:3037] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.198560 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8151ac615c84417cab1b210a95a0f32c" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:47.198671 32122 raft_consensus.cc:3037] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.198969 29862 raft_consensus.cc:2454] T 8151ac615c84417cab1b210a95a0f32c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:47.199612 32189 leader_election.cc:302] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:47.199780 6618 raft_consensus.cc:2781] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:47.199784 32122 raft_consensus.cc:2454] T 8151ac615c84417cab1b210a95a0f32c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:47.199859 6618 raft_consensus.cc:686] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:47.199980 6618 consensus_queue.cc:237] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.202203 29323 catalog_manager.cc:5484] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:47.208527 29470 consensus_queue.cc:237] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.209894 29600 raft_consensus.cc:1261] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:47.209893 31990 raft_consensus.cc:1261] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:47.210309 6543 consensus_queue.cc:1035] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:47.210510 5867 consensus_queue.cc:1035] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:47.212399 6543 raft_consensus.cc:2932] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.212713 29600 raft_consensus.cc:2932] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.214735 31990 raft_consensus.cc:2932] T 8eb9e9de44a046ffb4594755f44f15b8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.215561 29324 catalog_manager.cc:5484] T 8eb9e9de44a046ffb4594755f44f15b8 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.219112 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9e9de44a046ffb4594755f44f15b8 with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:47.219789 6726 tablet_replica.cc:330] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:47.219983 6726 raft_consensus.cc:2229] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:47.220131 6726 raft_consensus.cc:2258] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.220594 6726 ts_tablet_manager.cc:1875] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:47.223227 6726 ts_tablet_manager.cc:1888] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:47.223382 6726 log.cc:1196] T 8eb9e9de44a046ffb4594755f44f15b8 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/8eb9e9de44a046ffb4594755f44f15b8
I20240502 14:00:47.223933 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 8eb9e9de44a046ffb4594755f44f15b8 (table rebalancer_extra_table_8 [id=5991df84fefc4451a91456ea1ca247ed]) successfully deleted
I20240502 14:00:47.286437 6392 raft_consensus.cc:482] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:47.286579 6392 raft_consensus.cc:504] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.287125 6392 leader_election.cc:288] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:47.288496 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e59a9a97d9411e963d470b235706bd" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:47.288538 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e59a9a97d9411e963d470b235706bd" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:47.288645 32122 raft_consensus.cc:2454] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:47.288744 29470 raft_consensus.cc:2454] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:47.289086 31925 leader_election.cc:302] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:47.289741 6392 raft_consensus.cc:2781] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:47.289801 6392 raft_consensus.cc:482] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:47.289862 6392 raft_consensus.cc:3037] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.291133 6392 raft_consensus.cc:504] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.291525 6392 leader_election.cc:288] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:47.292058 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:47.292131 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e59a9a97d9411e963d470b235706bd" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:47.292138 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70e59a9a97d9411e963d470b235706bd" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:47.292251 32122 raft_consensus.cc:3037] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.292251 29470 raft_consensus.cc:3037] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.293285 29470 raft_consensus.cc:2454] T 70e59a9a97d9411e963d470b235706bd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:47.293285 32122 raft_consensus.cc:2454] T 70e59a9a97d9411e963d470b235706bd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:47.294389 31925 leader_election.cc:302] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:47.295107 6392 raft_consensus.cc:2781] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:47.295209 6392 raft_consensus.cc:686] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:47.295339 6392 consensus_queue.cc:237] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.298211 29325 catalog_manager.cc:5484] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:47.300555 6708 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:47.300889 6708 tablet.cc:1620] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:47.303705 6707 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:47.304000 6707 tablet.cc:1620] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:47.306934 6710 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:47.307236 6710 tablet.cc:1620] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:47.315511 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae] alter complete (version 2)
I20240502 14:00:47.359997 29582 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:47.360505 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:47.367259 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:47.367637 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:47.370599 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:47.372702 31839 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:47.381206 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:47.383234 29713 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:47.419800 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:47.427577 6708 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:47.428643 6707 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:47.432188 6710 tablet.cc:1722] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:47.437153 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae] alter complete (version 3)
I20240502 14:00:47.460783 29471 consensus_queue.cc:237] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.461954 31859 raft_consensus.cc:1261] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:47.461966 29601 raft_consensus.cc:1261] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:47.462373 6725 consensus_queue.cc:1035] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:47.462568 5867 consensus_queue.cc:1035] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:47.464231 6543 raft_consensus.cc:2932] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.464622 31859 raft_consensus.cc:2932] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.464679 29601 raft_consensus.cc:2932] T 1b37defc84d34af08f5744bf60a038c5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.468441 29328 catalog_manager.cc:5484] T 1b37defc84d34af08f5744bf60a038c5 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.472139 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b37defc84d34af08f5744bf60a038c5 with delete_type TABLET_DATA_TOMBSTONED (TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:47.472501 6799 tablet_replica.cc:330] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:47.472702 6799 raft_consensus.cc:2229] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:47.472827 6799 raft_consensus.cc:2258] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.473541 6799 ts_tablet_manager.cc:1875] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:47.476274 6799 ts_tablet_manager.cc:1888] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:47.476429 6799 log.cc:1196] T 1b37defc84d34af08f5744bf60a038c5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/1b37defc84d34af08f5744bf60a038c5
I20240502 14:00:47.476977 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 1b37defc84d34af08f5744bf60a038c5 (table rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]) successfully deleted
I20240502 14:00:47.542196 29321 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:47.542363 29321 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:47.544768 29321 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 9076cc42e8694807a0af8b0851b17935
I20240502 14:00:47.545581 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet 9076cc42e8694807a0af8b0851b17935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:47 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:47.545763 6726 tablet_replica.cc:330] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:47.545887 6726 raft_consensus.cc:2229] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:47.546062 6726 raft_consensus.cc:2258] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.547232 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 9076cc42e8694807a0af8b0851b17935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:47 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:47.547960 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 9076cc42e8694807a0af8b0851b17935 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:47 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:47.548066 6726 ts_tablet_manager.cc:1875] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:47.548300 6843 tablet_replica.cc:330] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:47.548493 6843 raft_consensus.cc:2229] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:47.548733 6843 raft_consensus.cc:2258] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.550339 6726 ts_tablet_manager.cc:1888] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:47.551113 6726 log.cc:1196] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/9076cc42e8694807a0af8b0851b17935
I20240502 14:00:47.551470 6726 ts_tablet_manager.cc:1909] T 9076cc42e8694807a0af8b0851b17935 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:47.552740 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 9076cc42e8694807a0af8b0851b17935 (table rebalancer_dynamic_table_new_name [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]) successfully deleted
I20240502 14:00:47.559775 6842 tablet_replica.cc:330] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:47.560024 6842 raft_consensus.cc:2229] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:47.560214 6842 raft_consensus.cc:2258] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.560928 6842 ts_tablet_manager.cc:1875] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:47.550097 6843 ts_tablet_manager.cc:1875] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:47.563315 6842 ts_tablet_manager.cc:1888] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:47.563428 6842 log.cc:1196] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/9076cc42e8694807a0af8b0851b17935
I20240502 14:00:47.563756 6842 ts_tablet_manager.cc:1909] T 9076cc42e8694807a0af8b0851b17935 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:47.564141 6843 ts_tablet_manager.cc:1888] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:47.564270 6843 log.cc:1196] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/9076cc42e8694807a0af8b0851b17935
I20240502 14:00:47.564496 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 9076cc42e8694807a0af8b0851b17935 (table rebalancer_dynamic_table_new_name [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]) successfully deleted
I20240502 14:00:47.566309 6843 ts_tablet_manager.cc:1909] T 9076cc42e8694807a0af8b0851b17935 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:47.567098 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 9076cc42e8694807a0af8b0851b17935 (table rebalancer_dynamic_table_new_name [id=e0cf49be1cde4c3fb02b1f9f1ae3c6ae]) successfully deleted
I20240502 14:00:47.587946 5985 consensus_queue.cc:1035] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.593297 6618 consensus_queue.cc:1035] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.619776 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:47.624648 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:47.625150 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:47.628378 29713 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:47.637676 29451 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:47.645349 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:47.649559 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:47.651285 29582 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:47.657238 29451 tablet_service.cc:1467] Processing CreateTablet for tablet fe0d9d655091441082cfe74e52599892 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5]), partition=
I20240502 14:00:47.657238 31839 tablet_service.cc:1467] Processing CreateTablet for tablet fe0d9d655091441082cfe74e52599892 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5]), partition=
I20240502 14:00:47.657554 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0d9d655091441082cfe74e52599892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.657573 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0d9d655091441082cfe74e52599892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.660712 31972 tablet_service.cc:1467] Processing CreateTablet for tablet fe0d9d655091441082cfe74e52599892 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5]), partition=
I20240502 14:00:47.660975 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0d9d655091441082cfe74e52599892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.671461 3671 consensus_queue.cc:1035] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.680899 3671 consensus_queue.cc:1035] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.691606 6860 tablet_bootstrap.cc:492] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:47.696828 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:47.697144 6859 tablet_bootstrap.cc:492] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:47.697494 6861 tablet_bootstrap.cc:492] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:47.702152 6860 tablet_bootstrap.cc:654] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.704893 6859 tablet_bootstrap.cc:654] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.705624 6861 tablet_bootstrap.cc:654] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.707471 6860 tablet_bootstrap.cc:492] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:47.710414 6859 tablet_bootstrap.cc:492] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:47.710942 6860 ts_tablet_manager.cc:1367] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 14:00:47.711556 6860 raft_consensus.cc:348] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.711838 6859 ts_tablet_manager.cc:1367] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20240502 14:00:47.712453 6859 raft_consensus.cc:348] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.716440 6860 raft_consensus.cc:374] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.717576 6860 raft_consensus.cc:729] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.718143 6860 consensus_queue.cc:260] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.718454 6859 raft_consensus.cc:374] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.718557 6859 raft_consensus.cc:729] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.719189 6860 ts_tablet_manager.cc:1398] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:00:47.718777 6859 consensus_queue.cc:260] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.722826 6859 ts_tablet_manager.cc:1398] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 14:00:47.727290 6861 tablet_bootstrap.cc:492] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:47.727502 6861 ts_tablet_manager.cc:1367] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240502 14:00:47.728816 6861 raft_consensus.cc:348] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.729117 6861 raft_consensus.cc:374] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.729216 6861 raft_consensus.cc:729] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.730075 6861 consensus_queue.cc:260] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.733960 6861 ts_tablet_manager.cc:1398] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:47.744485 6392 consensus_queue.cc:1035] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.754597 6487 consensus_queue.cc:1035] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.793267 32255 consensus_queue.cc:237] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.794982 29601 raft_consensus.cc:1261] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:47.795400 29471 raft_consensus.cc:1261] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:47.795408 5985 consensus_queue.cc:1035] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:47.795746 6618 consensus_queue.cc:1035] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:47.799517 5985 raft_consensus.cc:2932] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.800478 29471 raft_consensus.cc:2932] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.801906 29600 raft_consensus.cc:2932] T 19d48f827a6243448d344ee44a54482d P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.804028 29321 catalog_manager.cc:5484] T 19d48f827a6243448d344ee44a54482d P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:47.808048 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d48f827a6243448d344ee44a54482d with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:47.808286 6726 tablet_replica.cc:330] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:47.808446 6726 raft_consensus.cc:2229] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:47.808574 6726 raft_consensus.cc:2258] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.809026 6726 ts_tablet_manager.cc:1875] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:47.811291 6726 ts_tablet_manager.cc:1888] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:47.811371 6726 log.cc:1196] T 19d48f827a6243448d344ee44a54482d P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/19d48f827a6243448d344ee44a54482d
I20240502 14:00:47.811501 6543 raft_consensus.cc:482] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:47.811607 6543 raft_consensus.cc:504] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.811861 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 19d48f827a6243448d344ee44a54482d (table rebalancer_extra_table_0 [id=1d3a43f65e3c4e93951676b8a1bc2e6e]) successfully deleted
I20240502 14:00:47.811978 6543 leader_election.cc:288] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:47.812417 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0d9d655091441082cfe74e52599892" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:47.812441 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0d9d655091441082cfe74e52599892" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:47.812541 31860 raft_consensus.cc:2454] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:47.812572 31990 raft_consensus.cc:2454] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:47.813203 29404 leader_election.cc:302] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:47.813524 6543 raft_consensus.cc:2781] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:47.813584 6543 raft_consensus.cc:482] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:47.813620 6543 raft_consensus.cc:3037] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.814689 6543 raft_consensus.cc:504] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.815074 6543 leader_election.cc:288] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:47.815529 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0d9d655091441082cfe74e52599892" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:47.815682 31990 raft_consensus.cc:3037] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.816895 31990 raft_consensus.cc:2454] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:47.816946 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0d9d655091441082cfe74e52599892" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:47.817062 31860 raft_consensus.cc:3037] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.817832 31860 raft_consensus.cc:2454] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:47.818305 29404 leader_election.cc:302] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:47.818658 6543 raft_consensus.cc:2781] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:47.818763 6543 raft_consensus.cc:686] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:47.818961 6543 consensus_queue.cc:237] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:47.819743 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:47.821786 29323 catalog_manager.cc:5484] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:47.828744 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 1c91754a477542e5abea67a09971d111 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.829094 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c91754a477542e5abea67a09971d111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.829818 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea779bcca6749e196a4ed3ad8772a19 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.830163 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea779bcca6749e196a4ed3ad8772a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.831441 6901 tablet_bootstrap.cc:492] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:47.831418 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea779bcca6749e196a4ed3ad8772a19 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.831663 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea779bcca6749e196a4ed3ad8772a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.833020 6902 tablet_bootstrap.cc:492] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:47.833499 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea779bcca6749e196a4ed3ad8772a19 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.833760 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea779bcca6749e196a4ed3ad8772a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.833738 29451 tablet_service.cc:1467] Processing CreateTablet for tablet ef09c96ab4314611a0d51383ad31249b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.833925 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef09c96ab4314611a0d51383ad31249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.834419 6902 tablet_bootstrap.cc:654] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.836159 6860 tablet_bootstrap.cc:492] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:47.837442 6860 tablet_bootstrap.cc:654] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.837435 6902 tablet_bootstrap.cc:492] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:47.837570 6902 ts_tablet_manager.cc:1367] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:47.838297 6901 tablet_bootstrap.cc:654] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.838462 6902 raft_consensus.cc:348] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.838598 6902 raft_consensus.cc:374] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.838634 6902 raft_consensus.cc:729] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.838761 6902 consensus_queue.cc:260] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.840420 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 1c91754a477542e5abea67a09971d111 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.840692 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c91754a477542e5abea67a09971d111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.841078 6902 ts_tablet_manager.cc:1398] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:47.840896 31839 tablet_service.cc:1467] Processing CreateTablet for tablet ef09c96ab4314611a0d51383ad31249b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.841445 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef09c96ab4314611a0d51383ad31249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.843122 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 1c91754a477542e5abea67a09971d111 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.843420 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c91754a477542e5abea67a09971d111. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.843741 6859 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:47.844295 29713 tablet_service.cc:1467] Processing CreateTablet for tablet ef09c96ab4314611a0d51383ad31249b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:47.844558 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef09c96ab4314611a0d51383ad31249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:47.844810 6859 tablet_bootstrap.cc:654] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.846246 6859 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:47.846334 6859 ts_tablet_manager.cc:1367] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:47.847059 6859 raft_consensus.cc:348] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.847182 6859 raft_consensus.cc:374] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.847222 6859 raft_consensus.cc:729] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.847338 6859 consensus_queue.cc:260] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.847632 6859 ts_tablet_manager.cc:1398] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:47.847754 6909 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:47.848603 6909 tablet_bootstrap.cc:654] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.849654 6906 tablet_bootstrap.cc:492] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:47.852816 6906 tablet_bootstrap.cc:654] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.853412 6901 tablet_bootstrap.cc:492] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:47.853631 6901 ts_tablet_manager.cc:1367] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20240502 14:00:47.853972 6906 tablet_bootstrap.cc:492] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:47.854059 6906 ts_tablet_manager.cc:1367] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:47.854257 6901 raft_consensus.cc:348] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:47.854372 6901 raft_consensus.cc:374] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.854457 6901 raft_consensus.cc:729] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.854496 6906 raft_consensus.cc:348] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:47.854616 6906 raft_consensus.cc:374] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.854661 6906 raft_consensus.cc:729] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.854794 6906 consensus_queue.cc:260] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:47.855165 6906 ts_tablet_manager.cc:1398] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:47.857913 6901 consensus_queue.cc:260] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:47.858403 6901 ts_tablet_manager.cc:1398] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:47.858711 6901 tablet_bootstrap.cc:492] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:47.859711 6901 tablet_bootstrap.cc:654] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.861407 6908 tablet_bootstrap.cc:492] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:47.863221 6860 tablet_bootstrap.cc:492] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:47.863343 6860 ts_tablet_manager.cc:1367] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20240502 14:00:47.863957 6860 raft_consensus.cc:348] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.864092 6860 raft_consensus.cc:374] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.864132 6860 raft_consensus.cc:729] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.864279 6860 consensus_queue.cc:260] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.864650 6860 ts_tablet_manager.cc:1398] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:47.864850 6860 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:47.869112 6909 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:47.869230 6909 ts_tablet_manager.cc:1367] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240502 14:00:47.869288 6901 tablet_bootstrap.cc:492] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:47.869374 6901 ts_tablet_manager.cc:1367] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:47.869740 6909 raft_consensus.cc:348] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.869880 6909 raft_consensus.cc:374] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.869782 6901 raft_consensus.cc:348] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.869913 6909 raft_consensus.cc:729] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.869925 6901 raft_consensus.cc:374] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.869962 6901 raft_consensus.cc:729] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.870036 6909 consensus_queue.cc:260] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.870085 6901 consensus_queue.cc:260] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:47.870322 6909 ts_tablet_manager.cc:1398] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:47.870383 6901 ts_tablet_manager.cc:1398] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:47.880407 6860 tablet_bootstrap.cc:654] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.882633 6908 tablet_bootstrap.cc:654] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:47.883107 6860 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:47.883529 6860 ts_tablet_manager.cc:1367] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20240502 14:00:47.883993 6860 raft_consensus.cc:348] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.884105 6860 raft_consensus.cc:374] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.884145 6860 raft_consensus.cc:729] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.884132 6908 tablet_bootstrap.cc:492] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:47.884280 6860 consensus_queue.cc:260] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.884361 6908 ts_tablet_manager.cc:1367] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240502 14:00:47.884572 6860 ts_tablet_manager.cc:1398] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:47.885205 6908 raft_consensus.cc:348] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:47.885563 6908 raft_consensus.cc:374] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:47.885630 6908 raft_consensus.cc:729] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:47.885810 6908 consensus_queue.cc:260] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:47.887257 6908 ts_tablet_manager.cc:1398] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:47.927852 4741 raft_consensus.cc:482] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:47.927987 4741 raft_consensus.cc:504] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.928489 4741 leader_election.cc:288] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:47.929795 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09c96ab4314611a0d51383ad31249b" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:47.930004 29471 raft_consensus.cc:2454] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:47.930382 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09c96ab4314611a0d51383ad31249b" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:47.930529 31860 raft_consensus.cc:2454] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:47.930577 29668 leader_election.cc:302] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:47.930753 4741 raft_consensus.cc:2781] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:47.930795 4741 raft_consensus.cc:482] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:47.930827 4741 raft_consensus.cc:3037] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.932215 4741 raft_consensus.cc:504] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.932686 4741 leader_election.cc:288] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:47.933379 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09c96ab4314611a0d51383ad31249b" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:47.933506 31860 raft_consensus.cc:3037] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.933944 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09c96ab4314611a0d51383ad31249b" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:47.934100 29470 raft_consensus.cc:3037] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:47.934607 31860 raft_consensus.cc:2454] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:47.935160 29666 leader_election.cc:302] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:47.935425 4741 raft_consensus.cc:2781] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:47.935412 29470 raft_consensus.cc:2454] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:47.935523 4741 raft_consensus.cc:686] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:47.935653 4741 consensus_queue.cc:237] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:47.938364 29862 consensus_queue.cc:237] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:47.938789 29323 catalog_manager.cc:5484] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:47.940135 29729 raft_consensus.cc:1261] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:47.941118 5620 consensus_queue.cc:1035] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:47.942003 29601 raft_consensus.cc:1261] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:47.942363 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7650009dd8954f11ab2ecafef765692c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:47.942612 5620 consensus_queue.cc:1035] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:47.944794 5620 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:47.945302 29601 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:47.948618 29323 catalog_manager.cc:5484] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:47.949236 29729 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:47.952196 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:47.959268 31990 raft_consensus.cc:1261] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:47.959431 31860 raft_consensus.cc:1261] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:47.959710 6543 consensus_queue.cc:1035] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.960801 6725 consensus_queue.cc:1035] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:47.963599 29323 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet fe0d9d655091441082cfe74e52599892 (table rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5]). Expected version 1 got 0
I20240502 14:00:47.965265 6957 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:47.965620 6957 tablet.cc:1620] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:47.970788 29323 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet fe0d9d655091441082cfe74e52599892 (table rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5]). Expected version 1 got 0
I20240502 14:00:47.974960 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5] alter complete (version 1)
I20240502 14:00:47.979950 6956 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:47.980409 6956 tablet.cc:1620] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:47.981457 29323 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet fe0d9d655091441082cfe74e52599892 (table rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5]). Expected version 1 got 0
I20240502 14:00:47.991168 6960 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:47.991559 6960 tablet.cc:1620] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:48.015488 31992 consensus_queue.cc:237] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:48.017192 32255 raft_consensus.cc:1261] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:48.017585 31860 raft_consensus.cc:1261] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:48.017624 6972 consensus_queue.cc:1035] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:00:48.018355 31926 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e574c148957f44bf9c2935e955de99aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:48.018508 6487 consensus_queue.cc:1035] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:48.021806 6487 raft_consensus.cc:2932] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.022271 32255 raft_consensus.cc:2932] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.025529 29323 catalog_manager.cc:5484] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.027045 31860 raft_consensus.cc:2932] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.031332 6725 raft_consensus.cc:482] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:48.031448 6725 raft_consensus.cc:504] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.031961 6725 leader_election.cc:288] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:48.032539 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea779bcca6749e196a4ed3ad8772a19" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:48.032680 32122 raft_consensus.cc:2454] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:48.032749 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea779bcca6749e196a4ed3ad8772a19" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:48.032864 29601 raft_consensus.cc:2454] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:48.032984 29404 leader_election.cc:302] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:48.033473 6725 raft_consensus.cc:2781] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:48.033531 6725 raft_consensus.cc:482] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:48.033569 6725 raft_consensus.cc:3037] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.034755 6725 raft_consensus.cc:504] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.035131 6725 leader_election.cc:288] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:48.035984 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea779bcca6749e196a4ed3ad8772a19" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:48.036093 29601 raft_consensus.cc:3037] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.037034 29601 raft_consensus.cc:2454] T 5ea779bcca6749e196a4ed3ad8772a19 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:48.037065 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea779bcca6749e196a4ed3ad8772a19" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:48.037184 32122 raft_consensus.cc:3037] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.037976 32122 raft_consensus.cc:2454] T 5ea779bcca6749e196a4ed3ad8772a19 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:48.038291 29404 leader_election.cc:302] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:48.038445 6725 raft_consensus.cc:2781] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:48.038537 6725 raft_consensus.cc:686] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:48.038668 6725 consensus_queue.cc:237] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.041438 29321 catalog_manager.cc:5484] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:48.054656 5680 raft_consensus.cc:482] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:48.054793 5680 raft_consensus.cc:504] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:48.055361 5680 leader_election.cc:288] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:48.055923 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c91754a477542e5abea67a09971d111" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:48.055912 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c91754a477542e5abea67a09971d111" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:48.056066 29862 raft_consensus.cc:2454] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:48.056064 32255 raft_consensus.cc:2454] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:48.056375 29536 leader_election.cc:302] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:48.056564 5680 raft_consensus.cc:2781] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:48.056607 5680 raft_consensus.cc:482] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:48.056639 5680 raft_consensus.cc:3037] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.057773 5680 raft_consensus.cc:504] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:48.058115 5680 leader_election.cc:288] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:48.058405 29862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c91754a477542e5abea67a09971d111" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:48.058506 29862 raft_consensus.cc:3037] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.058591 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c91754a477542e5abea67a09971d111" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:48.058719 32255 raft_consensus.cc:3037] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.059530 29862 raft_consensus.cc:2454] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:48.059628 32255 raft_consensus.cc:2454] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:48.059815 29536 leader_election.cc:302] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:48.059975 5680 raft_consensus.cc:2781] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:48.060060 5680 raft_consensus.cc:686] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:48.060200 5680 consensus_queue.cc:237] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:48.062536 29321 catalog_manager.cc:5484] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:48.071879 29732 consensus_queue.cc:237] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.073730 29862 raft_consensus.cc:1261] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:48.073841 29601 raft_consensus.cc:1261] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:48.074268 6995 consensus_queue.cc:1035] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:48.074462 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09ce1f1d33dd4beca86a484a6f498175. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:48.074656 4741 consensus_queue.cc:1035] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:48.076326 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:48.076617 6288 raft_consensus.cc:2932] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:48.078693 29601 raft_consensus.cc:2932] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:48.079212 29862 raft_consensus.cc:2932] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:48.084647 29328 catalog_manager.cc:5484] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:48.085112 6956 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:48.085471 6956 tablet.cc:1620] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:48.086647 6957 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:48.087020 6957 tablet.cc:1620] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:48.089687 6960 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:48.090003 6960 tablet.cc:1620] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:48.094769 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=74c30d27a7e84b0bba4d676035002fd5] alter complete (version 2)
I20240502 14:00:48.111910 31992 consensus_queue.cc:237] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:48.113325 32255 raft_consensus.cc:1261] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:48.113401 29471 raft_consensus.cc:1261] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:48.113562 31926 consensus_peers.cc:485] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5edd4906c0914f6c83dc2bb1501d15bc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:48.113729 6976 consensus_queue.cc:1035] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:48.113883 6975 consensus_queue.cc:1035] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:48.115576 6976 raft_consensus.cc:2932] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.115973 32255 raft_consensus.cc:2932] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.117519 29471 raft_consensus.cc:2932] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.119107 29321 catalog_manager.cc:5484] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:48.198612 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:48.205366 6956 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:48.206065 6960 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:48.206617 6957 tablet.cc:1722] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:48.218062 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=74c30d27a7e84b0bba4d676035002fd5] alter complete (version 3)
I20240502 14:00:48.252729 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:48.253015 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:48.256183 29580 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:48.260390 29713 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:48.263047 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:48.269661 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:48.278303 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:48.279479 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:48.320950 29321 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:48.321171 29321 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:48.323732 29321 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet fe0d9d655091441082cfe74e52599892
I20240502 14:00:48.324580 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0d9d655091441082cfe74e52599892 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:48 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:48.331162 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0d9d655091441082cfe74e52599892 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:48 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:48.331578 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0d9d655091441082cfe74e52599892 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:48 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:48.331894 7073 tablet_replica.cc:330] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:48.341035 7073 raft_consensus.cc:2229] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:48.341315 7073 raft_consensus.cc:2258] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:48.342175 7073 ts_tablet_manager.cc:1875] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:48.344887 7073 ts_tablet_manager.cc:1888] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:48.345048 7073 log.cc:1196] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/fe0d9d655091441082cfe74e52599892
I20240502 14:00:48.345405 7073 ts_tablet_manager.cc:1909] T fe0d9d655091441082cfe74e52599892 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:48.346449 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet fe0d9d655091441082cfe74e52599892 (table rebalancer_dynamic_table_new_name [id=74c30d27a7e84b0bba4d676035002fd5]) successfully deleted
I20240502 14:00:48.347077 7071 tablet_replica.cc:330] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:48.347995 7071 raft_consensus.cc:2229] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:48.348284 7071 raft_consensus.cc:2258] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:48.349264 7071 ts_tablet_manager.cc:1875] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:48.355350 7072 tablet_replica.cc:330] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:48.355576 7072 raft_consensus.cc:2229] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:48.356159 7071 ts_tablet_manager.cc:1888] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:48.356382 7072 raft_consensus.cc:2258] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:48.356588 7071 log.cc:1196] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe0d9d655091441082cfe74e52599892
I20240502 14:00:48.357021 7071 ts_tablet_manager.cc:1909] T fe0d9d655091441082cfe74e52599892 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:48.357125 7072 ts_tablet_manager.cc:1875] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:48.357868 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet fe0d9d655091441082cfe74e52599892 (table rebalancer_dynamic_table_new_name [id=74c30d27a7e84b0bba4d676035002fd5]) successfully deleted
I20240502 14:00:48.360949 7072 ts_tablet_manager.cc:1888] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:48.363827 7072 log.cc:1196] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/fe0d9d655091441082cfe74e52599892
I20240502 14:00:48.364113 7072 ts_tablet_manager.cc:1909] T fe0d9d655091441082cfe74e52599892 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:48.365192 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet fe0d9d655091441082cfe74e52599892 (table rebalancer_dynamic_table_new_name [id=74c30d27a7e84b0bba4d676035002fd5]) successfully deleted
I20240502 14:00:48.420771 6998 consensus_queue.cc:1035] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.426698 4741 consensus_queue.cc:1035] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.432296 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:48.440783 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 9b80117e48e242729882a4fcc44bedb7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5]), partition=
I20240502 14:00:48.440816 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 9b80117e48e242729882a4fcc44bedb7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5]), partition=
I20240502 14:00:48.441107 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b80117e48e242729882a4fcc44bedb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.441118 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b80117e48e242729882a4fcc44bedb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.444053 7098 tablet_bootstrap.cc:492] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:48.444396 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 9b80117e48e242729882a4fcc44bedb7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5]), partition=
I20240502 14:00:48.444661 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b80117e48e242729882a4fcc44bedb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.445348 7098 tablet_bootstrap.cc:654] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.446461 7098 tablet_bootstrap.cc:492] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:48.446543 7098 ts_tablet_manager.cc:1367] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:48.447649 7098 raft_consensus.cc:348] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.447774 7098 raft_consensus.cc:374] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.447811 7098 raft_consensus.cc:729] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.447945 7098 consensus_queue.cc:260] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.448298 7098 ts_tablet_manager.cc:1398] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:48.448571 7101 tablet_bootstrap.cc:492] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:48.449505 7101 tablet_bootstrap.cc:654] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.452491 7101 tablet_bootstrap.cc:492] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:48.452584 7101 ts_tablet_manager.cc:1367] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:48.455860 7101 raft_consensus.cc:348] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.456012 7101 raft_consensus.cc:374] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.456107 7101 raft_consensus.cc:729] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.456293 7101 consensus_queue.cc:260] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.456692 7101 ts_tablet_manager.cc:1398] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:48.463054 7100 tablet_bootstrap.cc:492] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:48.464285 7100 tablet_bootstrap.cc:654] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.467247 7124 ts_tablet_manager.cc:902] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:48.475962 7124 tablet_copy_client.cc:308] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:48.477440 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 7650009dd8954f11ab2ecafef765692c from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37167)
I20240502 14:00:48.477562 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 7650009dd8954f11ab2ecafef765692c from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37167: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-7650009dd8954f11ab2ecafef765692c
I20240502 14:00:48.477721 7100 tablet_bootstrap.cc:492] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:48.477839 7100 ts_tablet_manager.cc:1367] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:00:48.478327 7100 raft_consensus.cc:348] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.478453 7100 raft_consensus.cc:374] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.478484 7100 raft_consensus.cc:729] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.478574 29884 tablet_copy_source_session.cc:205] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:48.478610 7100 consensus_queue.cc:260] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.478989 7100 ts_tablet_manager.cc:1398] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:48.480340 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7650009dd8954f11ab2ecafef765692c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.483454 7124 tablet_copy_client.cc:783] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:48.483808 7124 tablet_copy_client.cc:647] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:48.484635 6411 consensus_queue.cc:1035] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.488576 7124 tablet_copy_client.cc:523] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:48.491238 7124 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:48.502501 5680 consensus_queue.cc:1035] T 1c91754a477542e5abea67a09971d111 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.530606 4741 raft_consensus.cc:482] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:48.530741 4741 raft_consensus.cc:504] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.531190 4741 leader_election.cc:288] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:48.535136 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b80117e48e242729882a4fcc44bedb7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:48.535307 31861 raft_consensus.cc:2454] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:48.535946 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b80117e48e242729882a4fcc44bedb7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:48.536092 31990 raft_consensus.cc:2454] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:48.538197 29666 leader_election.cc:302] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:48.539121 4741 raft_consensus.cc:2781] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:48.539196 4741 raft_consensus.cc:482] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:48.539245 4741 raft_consensus.cc:3037] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.540663 4741 raft_consensus.cc:504] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.541172 4741 leader_election.cc:288] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:48.543010 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b80117e48e242729882a4fcc44bedb7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:48.543169 31861 raft_consensus.cc:3037] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.544415 31861 raft_consensus.cc:2454] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:48.545164 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b80117e48e242729882a4fcc44bedb7" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:48.545289 31990 raft_consensus.cc:3037] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.546566 31990 raft_consensus.cc:2454] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:48.547075 29666 leader_election.cc:302] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:48.547462 4741 raft_consensus.cc:2781] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:48.547561 4741 raft_consensus.cc:686] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:48.547732 4741 consensus_queue.cc:237] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:48.551014 29323 catalog_manager.cc:5484] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:48.559057 7140 ts_tablet_manager.cc:902] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:48.559852 7140 tablet_copy_client.cc:308] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:48.560523 7142 ts_tablet_manager.cc:902] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:48.560498 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet e574c148957f44bf9c2935e955de99aa from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:46293)
I20240502 14:00:48.560599 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet e574c148957f44bf9c2935e955de99aa from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:46293: session id = efdbf7bdfe0b41bdadb1add053b09a7e-e574c148957f44bf9c2935e955de99aa
I20240502 14:00:48.561161 7142 tablet_copy_client.cc:308] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:48.561573 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 09ce1f1d33dd4beca86a484a6f498175 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:55687)
I20240502 14:00:48.561655 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 09ce1f1d33dd4beca86a484a6f498175 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:55687: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-09ce1f1d33dd4beca86a484a6f498175
I20240502 14:00:48.561734 32012 tablet_copy_source_session.cc:205] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:48.562747 29753 tablet_copy_source_session.cc:205] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:48.563860 7142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ce1f1d33dd4beca86a484a6f498175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.567283 7140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e574c148957f44bf9c2935e955de99aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.570997 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:48.572090 7140 tablet_copy_client.cc:783] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:48.572260 7140 tablet_copy_client.cc:647] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:48.573264 6543 consensus_queue.cc:1035] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.581785 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69badccfbc416f92d982cf394ea0cd (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.581939 7142 tablet_copy_client.cc:783] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:48.582108 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69badccfbc416f92d982cf394ea0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.582552 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69badccfbc416f92d982cf394ea0cd (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.582818 32234 tablet_service.cc:1467] Processing CreateTablet for tablet ed0c8871730c4e2ba9f1320381159331 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.582906 7142 tablet_copy_client.cc:647] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:48.582818 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69badccfbc416f92d982cf394ea0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.585335 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69badccfbc416f92d982cf394ea0cd (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.585633 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69badccfbc416f92d982cf394ea0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.585659 32101 tablet_service.cc:1467] Processing CreateTablet for tablet ed0c8871730c4e2ba9f1320381159331 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.585783 31839 tablet_service.cc:1467] Processing CreateTablet for tablet f716493d95f84740beb85c3bf4021ad0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.585906 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0c8871730c4e2ba9f1320381159331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.586045 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f716493d95f84740beb85c3bf4021ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.588304 7101 tablet_bootstrap.cc:492] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:48.589402 7101 tablet_bootstrap.cc:654] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.589999 7142 tablet_copy_client.cc:523] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:48.590296 31972 tablet_service.cc:1467] Processing CreateTablet for tablet f716493d95f84740beb85c3bf4021ad0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.590628 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f716493d95f84740beb85c3bf4021ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.590734 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0c8871730c4e2ba9f1320381159331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.592900 7100 tablet_bootstrap.cc:492] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:48.593497 7142 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:48.593976 7100 tablet_bootstrap.cc:654] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.596783 29451 tablet_service.cc:1467] Processing CreateTablet for tablet f716493d95f84740beb85c3bf4021ad0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.597070 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f716493d95f84740beb85c3bf4021ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.589727 7153 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:48.605473 7101 tablet_bootstrap.cc:492] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:48.605595 7101 ts_tablet_manager.cc:1367] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:48.606130 7101 raft_consensus.cc:348] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.606189 6725 consensus_queue.cc:1035] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.606261 7101 raft_consensus.cc:374] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.606292 7101 raft_consensus.cc:729] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.606401 7101 consensus_queue.cc:260] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.602332 7153 tablet_bootstrap.cc:654] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.606726 7101 ts_tablet_manager.cc:1398] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:48.609513 29843 tablet_service.cc:1467] Processing CreateTablet for tablet ed0c8871730c4e2ba9f1320381159331 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:48.609812 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0c8871730c4e2ba9f1320381159331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.616047 7140 tablet_copy_client.cc:523] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:48.617738 7140 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:48.620594 7154 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:48.624011 7140 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:48.624325 7140 tablet_bootstrap.cc:492] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:00:48.624351 7154 tablet_bootstrap.cc:654] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.624504 7140 ts_tablet_manager.cc:1367] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:00:48.625182 7140 raft_consensus.cc:348] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:48.625316 7140 raft_consensus.cc:729] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.625448 7140 consensus_queue.cc:260] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:48.625766 7140 ts_tablet_manager.cc:1398] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:48.626508 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-e574c148957f44bf9c2935e955de99aa received from {username='slave'} at 127.22.55.131:46293
I20240502 14:00:48.626583 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-e574c148957f44bf9c2935e955de99aa on tablet e574c148957f44bf9c2935e955de99aa with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:48.628540 7140 ts_tablet_manager.cc:902] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:48.629510 7100 tablet_bootstrap.cc:492] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:48.629614 7100 ts_tablet_manager.cc:1367] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240502 14:00:48.630206 7100 raft_consensus.cc:348] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.630328 7100 raft_consensus.cc:374] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.630362 7100 raft_consensus.cc:729] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.630483 7100 consensus_queue.cc:260] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.630790 7100 ts_tablet_manager.cc:1398] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:48.630906 7140 tablet_copy_client.cc:308] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:48.631188 7155 tablet_bootstrap.cc:492] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:48.632318 7155 tablet_bootstrap.cc:654] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.634513 7152 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:48.637173 7152 tablet_bootstrap.cc:654] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.637722 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet 5edd4906c0914f6c83dc2bb1501d15bc from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:46293)
I20240502 14:00:48.637835 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet 5edd4906c0914f6c83dc2bb1501d15bc from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:46293: session id = efdbf7bdfe0b41bdadb1add053b09a7e-5edd4906c0914f6c83dc2bb1501d15bc
I20240502 14:00:48.639024 32012 tablet_copy_source_session.cc:205] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:48.639041 7153 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:48.639140 7153 ts_tablet_manager.cc:1367] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240502 14:00:48.640231 7161 tablet_bootstrap.cc:492] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:48.640321 7153 raft_consensus.cc:348] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.641034 7153 raft_consensus.cc:374] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.641165 7153 raft_consensus.cc:729] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.641778 7161 tablet_bootstrap.cc:654] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.641929 7153 consensus_queue.cc:260] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.643877 7153 ts_tablet_manager.cc:1398] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:48.644166 7153 tablet_bootstrap.cc:492] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:48.646230 7153 tablet_bootstrap.cc:654] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.646819 7140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edd4906c0914f6c83dc2bb1501d15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:48.649601 7154 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:48.649783 7154 ts_tablet_manager.cc:1367] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 14:00:48.650580 7154 raft_consensus.cc:348] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.650806 7154 raft_consensus.cc:374] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.650930 7154 raft_consensus.cc:729] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.651252 7154 consensus_queue.cc:260] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.652915 7154 ts_tablet_manager.cc:1398] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:48.654794 7153 tablet_bootstrap.cc:492] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:48.654978 7153 ts_tablet_manager.cc:1367] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 14:00:48.657775 7153 raft_consensus.cc:348] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.658015 7153 raft_consensus.cc:374] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.658113 7153 raft_consensus.cc:729] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.658305 7153 consensus_queue.cc:260] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.662583 7153 ts_tablet_manager.cc:1398] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:48.672653 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:48.677230 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:48.677660 32234 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:48.678622 29451 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:48.679837 31861 raft_consensus.cc:1261] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:48.680487 31990 raft_consensus.cc:1261] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:48.681172 4741 consensus_queue.cc:1035] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.681363 6998 consensus_queue.cc:1035] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:48.687960 29321 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet 9b80117e48e242729882a4fcc44bedb7 (table rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5]). Expected version 1 got 0
I20240502 14:00:48.689162 29321 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 9b80117e48e242729882a4fcc44bedb7 (table rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5]). Expected version 1 got 0
I20240502 14:00:48.693161 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:48.695994 7140 tablet_copy_client.cc:783] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:48.696200 7140 tablet_copy_client.cc:647] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:48.696895 7155 tablet_bootstrap.cc:492] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:48.697065 7155 ts_tablet_manager.cc:1367] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.000s
I20240502 14:00:48.698195 7155 raft_consensus.cc:348] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.698405 7155 raft_consensus.cc:374] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.698477 7155 raft_consensus.cc:729] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.698648 7155 consensus_queue.cc:260] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.699028 7155 ts_tablet_manager.cc:1398] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:48.703495 7152 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:48.703656 7152 ts_tablet_manager.cc:1367] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.000s
I20240502 14:00:48.704241 7152 raft_consensus.cc:348] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.704437 7152 raft_consensus.cc:374] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.704509 7152 raft_consensus.cc:729] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.704690 7152 consensus_queue.cc:260] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.705039 7152 ts_tablet_manager.cc:1398] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:48.715288 7161 tablet_bootstrap.cc:492] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:48.715494 7161 ts_tablet_manager.cc:1367] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.000s
I20240502 14:00:48.716075 7161 raft_consensus.cc:348] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.716272 7161 raft_consensus.cc:374] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.716356 7161 raft_consensus.cc:729] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.716570 7161 consensus_queue.cc:260] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.717401 7161 ts_tablet_manager.cc:1398] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:48.718585 29321 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet 9b80117e48e242729882a4fcc44bedb7 (table rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5]). Expected version 1 got 0
I20240502 14:00:48.721155 29321 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet 9b80117e48e242729882a4fcc44bedb7 (table rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5]). Expected version 1 got 0
I20240502 14:00:48.725700 7187 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:48.726207 29713 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:48.726982 7187 tablet.cc:1620] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:48.739461 7190 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:48.743712 7190 tablet.cc:1620] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:48.745105 7140 tablet_copy_client.cc:523] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:48.749038 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5] alter complete (version 1)
I20240502 14:00:48.751827 29580 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:48.758333 7140 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:48.763622 7142 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:48.764802 7142 tablet_bootstrap.cc:492] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:48.765163 7142 ts_tablet_manager.cc:1367] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.172s user 0.027s sys 0.004s
I20240502 14:00:48.765827 7142 raft_consensus.cc:348] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.767498 7142 raft_consensus.cc:729] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.770437 31839 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:48.770710 32101 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:48.771023 7152 tablet_bootstrap.cc:492] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:48.771783 7189 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:48.772104 7152 tablet_bootstrap.cc:654] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:48.772265 7189 tablet.cc:1620] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:48.767931 7142 consensus_queue.cc:260] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.773064 7124 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:48.773415 7124 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:48.773608 7124 ts_tablet_manager.cc:1367] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.282s user 0.043s sys 0.000s
I20240502 14:00:48.774081 7124 raft_consensus.cc:348] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.774197 7124 raft_consensus.cc:729] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.774323 7124 consensus_queue.cc:260] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.774642 7124 ts_tablet_manager.cc:1398] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:48.776219 7152 tablet_bootstrap.cc:492] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:48.776360 7152 ts_tablet_manager.cc:1367] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:48.776818 7152 raft_consensus.cc:348] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.776976 7152 raft_consensus.cc:374] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:48.777082 7152 raft_consensus.cc:729] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.777266 7152 consensus_queue.cc:260] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.777665 7152 ts_tablet_manager.cc:1398] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:48.790062 7140 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:48.790364 7140 tablet_bootstrap.cc:492] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:00:48.790544 7140 ts_tablet_manager.cc:1367] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20240502 14:00:48.791039 7140 raft_consensus.cc:348] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:48.791173 7140 raft_consensus.cc:729] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:48.791321 7140 consensus_queue.cc:260] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:48.791618 7140 ts_tablet_manager.cc:1398] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:00:48.792534 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-5edd4906c0914f6c83dc2bb1501d15bc received from {username='slave'} at 127.22.55.131:46293
I20240502 14:00:48.792619 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-5edd4906c0914f6c83dc2bb1501d15bc on tablet 5edd4906c0914f6c83dc2bb1501d15bc with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:48.792729 7142 ts_tablet_manager.cc:1398] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240502 14:00:48.802637 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-7650009dd8954f11ab2ecafef765692c received from {username='slave'} at 127.22.55.135:37167
I20240502 14:00:48.802747 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-7650009dd8954f11ab2ecafef765692c on tablet 7650009dd8954f11ab2ecafef765692c with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:48.803541 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-09ce1f1d33dd4beca86a484a6f498175 received from {username='slave'} at 127.22.55.135:55687
I20240502 14:00:48.803619 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-09ce1f1d33dd4beca86a484a6f498175 on tablet 09ce1f1d33dd4beca86a484a6f498175 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:48.858105 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:48.864370 7187 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:48.864712 7187 tablet.cc:1620] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: MemRowSet was empty: no flush needed.
I20240502 14:00:48.865296 7189 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:48.865619 7189 tablet.cc:1620] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:48.866127 7190 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:48.866425 7190 tablet.cc:1620] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:48.870708 6975 raft_consensus.cc:482] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:48.870808 6975 raft_consensus.cc:504] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.871243 6975 leader_election.cc:288] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:48.872017 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:48.872165 29471 raft_consensus.cc:2454] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:48.875716 5217 raft_consensus.cc:482] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:48.875811 5217 raft_consensus.cc:504] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.876241 5217 leader_election.cc:288] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:48.876614 32119 raft_consensus.cc:1203] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:48.876803 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c8871730c4e2ba9f1320381159331" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:48.876951 32255 raft_consensus.cc:2454] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:48.877372 32058 leader_election.cc:302] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:48.877703 31927 leader_election.cc:302] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:48.880246 6615 raft_consensus.cc:482] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:48.880332 6615 raft_consensus.cc:504] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.880703 6615 leader_election.cc:288] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:48.881232 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:48.881362 31990 raft_consensus.cc:2454] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:48.881830 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:48.881866 31794 leader_election.cc:302] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:48.881978 29471 raft_consensus.cc:2454] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:48.882032 6615 raft_consensus.cc:2781] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:48.882086 6615 raft_consensus.cc:482] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:48.882138 6615 raft_consensus.cc:3037] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.882493 6625 raft_consensus.cc:2781] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:48.882546 6625 raft_consensus.cc:482] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:48.882591 6625 raft_consensus.cc:3037] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.882688 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c8871730c4e2ba9f1320381159331" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:48.882817 29863 raft_consensus.cc:2454] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:48.883280 6615 raft_consensus.cc:504] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.883670 6615 leader_election.cc:288] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:48.883689 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:48.883685 6625 raft_consensus.cc:504] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.883806 31860 raft_consensus.cc:2379] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 269b366a09ff435ea8ec8dc01d0f9f6e in current term 1: Already voted for candidate f0e01c4f581843e9ad4ced97f0072dc4 in this term.
I20240502 14:00:48.883797 6972 raft_consensus.cc:2781] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:48.883846 6972 raft_consensus.cc:482] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:48.883883 6972 raft_consensus.cc:3037] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.884074 6625 leader_election.cc:288] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:48.884974 6972 raft_consensus.cc:504] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.885371 6972 leader_election.cc:288] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:48.885320 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:48.885447 29471 raft_consensus.cc:3037] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.885584 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c8871730c4e2ba9f1320381159331" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:48.885712 32254 raft_consensus.cc:3037] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.886096 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:48.886207 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:48.886255 31990 raft_consensus.cc:2379] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f0e01c4f581843e9ad4ced97f0072dc4 in current term 1: Already voted for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in this term.
I20240502 14:00:48.886510 29471 raft_consensus.cc:2454] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:48.886751 32254 raft_consensus.cc:2454] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:48.886864 31796 leader_election.cc:302] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:48.887142 6615 raft_consensus.cc:2781] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:48.887235 32058 leader_election.cc:302] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:48.887305 6615 raft_consensus.cc:686] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:48.887450 6615 consensus_queue.cc:237] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:48.887543 6625 raft_consensus.cc:2781] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:48.887681 6625 raft_consensus.cc:686] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:48.887707 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c8871730c4e2ba9f1320381159331" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:48.887806 29863 raft_consensus.cc:3037] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.887806 6625 consensus_queue.cc:237] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:48.888800 29863 raft_consensus.cc:2454] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:48.889264 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:48.889537 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=f505c5baa40642e4a6c0a55f6441d3e5] alter complete (version 2)
I20240502 14:00:48.889959 31925 leader_election.cc:302] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:48.891372 6975 raft_consensus.cc:2726] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:00:48.892037 29323 catalog_manager.cc:5484] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:48.896587 29323 catalog_manager.cc:5484] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:48.988569 7191 raft_consensus.cc:482] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:48.988744 7191 raft_consensus.cc:504] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.989131 7191 leader_election.cc:288] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:48.989483 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69badccfbc416f92d982cf394ea0cd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:48.989631 29863 raft_consensus.cc:2454] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:48.989617 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69badccfbc416f92d982cf394ea0cd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:48.989785 32254 raft_consensus.cc:2454] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:48.989938 29536 leader_election.cc:302] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:48.990361 6411 raft_consensus.cc:2781] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:48.990428 6411 raft_consensus.cc:482] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:48.990465 6411 raft_consensus.cc:3037] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.991499 6411 raft_consensus.cc:504] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.991811 6411 leader_election.cc:288] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:48.992132 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69badccfbc416f92d982cf394ea0cd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:48.992123 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69badccfbc416f92d982cf394ea0cd" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:48.992241 29864 raft_consensus.cc:3037] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.992241 32254 raft_consensus.cc:3037] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:48.993211 29864 raft_consensus.cc:2454] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:48.993211 32254 raft_consensus.cc:2454] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:48.993525 29536 leader_election.cc:302] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:48.993680 6411 raft_consensus.cc:2781] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:48.993764 6411 raft_consensus.cc:686] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:48.993929 6411 consensus_queue.cc:237] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:48.997140 29323 catalog_manager.cc:5484] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:48.998281 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:49.003422 7187 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:49.003821 7189 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:49.006103 7190 tablet.cc:1722] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:49.011088 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=f505c5baa40642e4a6c0a55f6441d3e5] alter complete (version 3)
I20240502 14:00:49.027004 29451 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:49.030552 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:49.036168 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:49.038240 32234 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:49.046286 29580 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:49.059451 31839 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:49.071801 29713 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:49.078795 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:49.093847 29732 raft_consensus.cc:1203] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:49.114490 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:49.114673 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:49.116816 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 9b80117e48e242729882a4fcc44bedb7
I20240502 14:00:49.117527 31839 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b80117e48e242729882a4fcc44bedb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:49 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:49.117771 31972 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b80117e48e242729882a4fcc44bedb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:49 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:49.117797 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b80117e48e242729882a4fcc44bedb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:49 UTC) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:49.118041 7257 tablet_replica.cc:330] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:49.118083 7258 tablet_replica.cc:330] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:49.118214 7255 tablet_replica.cc:330] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:49.118229 7258 raft_consensus.cc:2229] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:49.118228 7257 raft_consensus.cc:2229] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:49.118343 7257 raft_consensus.cc:2258] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.118446 7258 raft_consensus.cc:2258] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.118892 7255 raft_consensus.cc:2229] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:49.118942 7257 ts_tablet_manager.cc:1875] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:49.119042 7255 raft_consensus.cc:2258] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.119654 7258 ts_tablet_manager.cc:1875] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:49.119674 7255 ts_tablet_manager.cc:1875] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:49.122128 7258 ts_tablet_manager.cc:1888] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:49.122138 7255 ts_tablet_manager.cc:1888] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:49.122220 7255 log.cc:1196] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/9b80117e48e242729882a4fcc44bedb7
I20240502 14:00:49.122273 7258 log.cc:1196] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b80117e48e242729882a4fcc44bedb7
I20240502 14:00:49.122648 7255 ts_tablet_manager.cc:1909] T 9b80117e48e242729882a4fcc44bedb7 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:49.123170 7258 ts_tablet_manager.cc:1909] T 9b80117e48e242729882a4fcc44bedb7 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting consensus metadata
I20240502 14:00:49.123766 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 9b80117e48e242729882a4fcc44bedb7 (table rebalancer_dynamic_table_new_name [id=f505c5baa40642e4a6c0a55f6441d3e5]) successfully deleted
I20240502 14:00:49.124188 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 9b80117e48e242729882a4fcc44bedb7 (table rebalancer_dynamic_table_new_name [id=f505c5baa40642e4a6c0a55f6441d3e5]) successfully deleted
I20240502 14:00:49.134987 7257 ts_tablet_manager.cc:1888] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:49.135118 7257 log.cc:1196] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/9b80117e48e242729882a4fcc44bedb7
I20240502 14:00:49.135478 7257 ts_tablet_manager.cc:1909] T 9b80117e48e242729882a4fcc44bedb7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:49.136257 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 9b80117e48e242729882a4fcc44bedb7 (table rebalancer_dynamic_table_new_name [id=f505c5baa40642e4a6c0a55f6441d3e5]) successfully deleted
I20240502 14:00:49.169217 32119 raft_consensus.cc:1203] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:49.199678 29732 raft_consensus.cc:1203] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:49.219538 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:49.227118 31839 tablet_service.cc:1467] Processing CreateTablet for tablet 65c13043f597449cbe533eca8c44a060 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebaee5aa390046d7aba6ada87d472ecd]), partition=
I20240502 14:00:49.227393 32234 tablet_service.cc:1467] Processing CreateTablet for tablet 65c13043f597449cbe533eca8c44a060 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebaee5aa390046d7aba6ada87d472ecd]), partition=
I20240502 14:00:49.227456 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c13043f597449cbe533eca8c44a060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.227654 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c13043f597449cbe533eca8c44a060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.230645 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 65c13043f597449cbe533eca8c44a060 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebaee5aa390046d7aba6ada87d472ecd]), partition=
I20240502 14:00:49.230963 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c13043f597449cbe533eca8c44a060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.232321 7307 tablet_bootstrap.cc:492] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:49.234098 7307 tablet_bootstrap.cc:654] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.234934 7308 tablet_bootstrap.cc:492] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:49.237110 7308 tablet_bootstrap.cc:654] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.241540 7305 tablet_bootstrap.cc:492] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:49.241634 7308 tablet_bootstrap.cc:492] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:49.241739 7308 ts_tablet_manager.cc:1367] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:49.242206 7308 raft_consensus.cc:348] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.242372 7308 raft_consensus.cc:374] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.242410 7308 raft_consensus.cc:729] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.242547 7308 consensus_queue.cc:260] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.242868 7308 ts_tablet_manager.cc:1398] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:49.245242 7305 tablet_bootstrap.cc:654] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.247073 7305 tablet_bootstrap.cc:492] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:49.247253 7305 ts_tablet_manager.cc:1367] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:49.247745 7305 raft_consensus.cc:348] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.247934 7305 raft_consensus.cc:374] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.248023 7305 raft_consensus.cc:729] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.248194 7305 consensus_queue.cc:260] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.249423 7305 ts_tablet_manager.cc:1398] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:49.251682 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:49.255478 7307 tablet_bootstrap.cc:492] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:49.255635 7307 ts_tablet_manager.cc:1367] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:00:49.256186 7307 raft_consensus.cc:348] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.256307 7307 raft_consensus.cc:374] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.256335 7307 raft_consensus.cc:729] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.256500 7307 consensus_queue.cc:260] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.260143 7307 ts_tablet_manager.cc:1398] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:49.261387 29451 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:49.267866 29580 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:49.272089 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:49.272374 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:49.276840 29713 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:49.295243 31839 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:49.296808 32234 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:49.297233 7191 raft_consensus.cc:482] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:49.297341 7191 raft_consensus.cc:504] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.297736 7191 leader_election.cc:288] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:49.298035 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c13043f597449cbe533eca8c44a060" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:49.298175 32254 raft_consensus.cc:2454] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:49.298486 29537 leader_election.cc:302] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:49.298661 7191 raft_consensus.cc:2781] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:49.298715 7191 raft_consensus.cc:482] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:49.298755 7191 raft_consensus.cc:3037] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.299589 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c13043f597449cbe533eca8c44a060" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:49.299731 31860 raft_consensus.cc:2454] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:49.299996 7191 raft_consensus.cc:504] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.300351 7191 leader_election.cc:288] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:49.302215 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c13043f597449cbe533eca8c44a060" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:49.302332 31860 raft_consensus.cc:3037] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.302985 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c13043f597449cbe533eca8c44a060" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:49.303103 32255 raft_consensus.cc:3037] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.303371 31860 raft_consensus.cc:2454] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:49.303655 29535 leader_election.cc:302] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:49.304049 32255 raft_consensus.cc:2454] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:49.313056 7191 raft_consensus.cc:2781] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:49.313170 7191 raft_consensus.cc:686] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:49.313297 7191 consensus_queue.cc:237] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.316696 29321 catalog_manager.cc:5484] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:49.324978 7235 consensus_queue.cc:1035] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.352021 6614 consensus_queue.cc:1035] T f716493d95f84740beb85c3bf4021ad0 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.381049 5217 consensus_queue.cc:1035] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.381212 29864 consensus_queue.cc:237] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:49.383108 32119 raft_consensus.cc:1261] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:49.383692 7333 consensus_queue.cc:1035] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:49.383837 29601 raft_consensus.cc:1261] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:49.384183 6950 consensus_queue.cc:1035] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:49.386430 7237 consensus_queue.cc:1035] T ed0c8871730c4e2ba9f1320381159331 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.387914 7333 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:49.390252 29601 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:49.392269 29323 catalog_manager.cc:5484] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:49.396306 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 7650009dd8954f11ab2ecafef765692c with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:49.396678 7258 tablet_replica.cc:330] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:49.396833 7258 raft_consensus.cc:2229] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:49.396991 7258 raft_consensus.cc:2258] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.397562 7258 ts_tablet_manager.cc:1875] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:49.399574 7258 ts_tablet_manager.cc:1888] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:49.399652 7258 log.cc:1196] T 7650009dd8954f11ab2ecafef765692c P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/7650009dd8954f11ab2ecafef765692c
I20240502 14:00:49.399830 32119 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:49.401811 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 7650009dd8954f11ab2ecafef765692c (table rebalance_test_table_15 [id=8c338f82c9ff4eca8915abc0b21511db]) successfully deleted
I20240502 14:00:49.424362 29321 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:49.429280 32255 raft_consensus.cc:1261] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:49.430039 6411 consensus_queue.cc:1035] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.432471 31860 raft_consensus.cc:1261] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:49.433374 7359 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:49.433800 7359 tablet.cc:1620] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:49.434191 29323 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 65c13043f597449cbe533eca8c44a060 (table rebalancer_dynamic_table [id=ebaee5aa390046d7aba6ada87d472ecd]). Expected version 1 got 0
I20240502 14:00:49.434350 7360 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:49.434662 7360 tablet.cc:1620] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:49.434690 7191 consensus_queue.cc:1035] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.439384 7189 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:49.439711 7189 tablet.cc:1620] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:49.439802 29321 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet 65c13043f597449cbe533eca8c44a060 (table rebalancer_dynamic_table [id=ebaee5aa390046d7aba6ada87d472ecd]). Expected version 1 got 0
I20240502 14:00:49.441392 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=ebaee5aa390046d7aba6ada87d472ecd] alter complete (version 1)
I20240502 14:00:49.480748 6411 consensus_queue.cc:1035] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.484282 7191 consensus_queue.cc:1035] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:49.530095 31972 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:49.534615 29451 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:49.539340 32234 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:49.548148 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:49.550602 29327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:49.552244 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:49.553982 31839 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:49.565037 7359 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:49.565407 7359 tablet.cc:1620] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:49.565582 7360 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:49.565865 7360 tablet.cc:1620] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:49.568213 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 00d11f638cfa4b53b532c9084fa45610 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.568507 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d11f638cfa4b53b532c9084fa45610. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.569525 29450 tablet_service.cc:1467] Processing CreateTablet for tablet f28e9f5068aa48b5a87841bdf720d9e0 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.569818 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28e9f5068aa48b5a87841bdf720d9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.573611 7400 tablet_bootstrap.cc:492] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:49.573971 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=ebaee5aa390046d7aba6ada87d472ecd] alter complete (version 2)
I20240502 14:00:49.574133 29451 tablet_service.cc:1467] Processing CreateTablet for tablet fad144c461044c299ea905832908666a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.574390 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad144c461044c299ea905832908666a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.574918 7400 tablet_bootstrap.cc:654] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.575821 29713 tablet_service.cc:1467] Processing CreateTablet for tablet fad144c461044c299ea905832908666a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.575882 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 00d11f638cfa4b53b532c9084fa45610 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.576081 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad144c461044c299ea905832908666a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.576575 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d11f638cfa4b53b532c9084fa45610. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.576745 29580 tablet_service.cc:1467] Processing CreateTablet for tablet f28e9f5068aa48b5a87841bdf720d9e0 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.576993 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28e9f5068aa48b5a87841bdf720d9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.578382 29711 tablet_service.cc:1467] Processing CreateTablet for tablet f28e9f5068aa48b5a87841bdf720d9e0 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.578579 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28e9f5068aa48b5a87841bdf720d9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.579077 7402 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:49.579182 7308 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:49.580152 7308 tablet_bootstrap.cc:654] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.580230 7402 tablet_bootstrap.cc:654] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.580739 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:49.581146 31839 tablet_service.cc:1467] Processing CreateTablet for tablet fad144c461044c299ea905832908666a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.581390 7402 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:49.581419 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad144c461044c299ea905832908666a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.581458 7402 ts_tablet_manager.cc:1367] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:49.581863 7402 raft_consensus.cc:348] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.581957 7402 raft_consensus.cc:374] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.581983 7402 raft_consensus.cc:729] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.582084 7402 consensus_queue.cc:260] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.582319 7402 ts_tablet_manager.cc:1398] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:49.582466 7402 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:49.583484 7402 tablet_bootstrap.cc:654] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.583559 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 00d11f638cfa4b53b532c9084fa45610 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:49.583763 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d11f638cfa4b53b532c9084fa45610. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.585783 7307 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:49.586666 7307 tablet_bootstrap.cc:654] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.590540 7189 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:49.590895 7189 tablet.cc:1620] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:49.596596 7308 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:49.596707 7308 ts_tablet_manager.cc:1367] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:00:49.597201 7308 raft_consensus.cc:348] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.597329 7308 raft_consensus.cc:374] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.597366 7308 raft_consensus.cc:729] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.597502 7308 consensus_queue.cc:260] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.597811 7308 ts_tablet_manager.cc:1398] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:49.600211 29712 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:49.604727 7307 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:49.604842 7307 ts_tablet_manager.cc:1367] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240502 14:00:49.605365 7307 raft_consensus.cc:348] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.605497 7307 raft_consensus.cc:374] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.605525 7307 raft_consensus.cc:729] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.605625 7307 consensus_queue.cc:260] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.605872 7307 ts_tablet_manager.cc:1398] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:49.606026 7307 tablet_bootstrap.cc:492] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:49.610611 7307 tablet_bootstrap.cc:654] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.611771 7402 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:49.611939 7402 ts_tablet_manager.cc:1367] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:00:49.612818 7402 raft_consensus.cc:348] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.613022 7402 raft_consensus.cc:374] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.613116 7402 raft_consensus.cc:729] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.613303 7402 consensus_queue.cc:260] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.613734 7402 ts_tablet_manager.cc:1398] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:49.614713 7400 tablet_bootstrap.cc:492] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:49.614904 7400 ts_tablet_manager.cc:1367] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.001s
I20240502 14:00:49.615003 7307 tablet_bootstrap.cc:492] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:49.615089 7307 ts_tablet_manager.cc:1367] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:00:49.615465 7400 raft_consensus.cc:348] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.615515 7307 raft_consensus.cc:348] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.615576 7400 raft_consensus.cc:374] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.615608 7307 raft_consensus.cc:374] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.615607 7400 raft_consensus.cc:729] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.615641 7307 raft_consensus.cc:729] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.615751 7307 consensus_queue.cc:260] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.616246 7402 tablet_bootstrap.cc:492] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:49.616514 7400 consensus_queue.cc:260] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.617791 7307 ts_tablet_manager.cc:1398] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:49.618706 7402 tablet_bootstrap.cc:654] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.621812 7400 ts_tablet_manager.cc:1398] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:00:49.621883 7401 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:49.624198 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:49.625267 7401 tablet_bootstrap.cc:654] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.626351 7401 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:49.626452 7401 ts_tablet_manager.cc:1367] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:49.626893 7401 raft_consensus.cc:348] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.626997 7402 tablet_bootstrap.cc:492] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:49.627010 7401 raft_consensus.cc:374] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.627040 7401 raft_consensus.cc:729] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.627074 7402 ts_tablet_manager.cc:1367] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:49.627168 7401 consensus_queue.cc:260] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.627430 7401 ts_tablet_manager.cc:1398] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:49.627714 7402 raft_consensus.cc:348] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.627813 7402 raft_consensus.cc:374] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.627841 7402 raft_consensus.cc:729] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.627952 7402 consensus_queue.cc:260] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.628351 7402 ts_tablet_manager.cc:1398] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:49.630303 7401 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:49.632908 7401 tablet_bootstrap.cc:654] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.635516 7401 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:49.635593 7401 ts_tablet_manager.cc:1367] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:49.636013 7401 raft_consensus.cc:348] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.636118 7401 raft_consensus.cc:374] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.636147 7401 raft_consensus.cc:729] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.636271 7401 consensus_queue.cc:260] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.636541 7401 ts_tablet_manager.cc:1398] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:49.669819 29601 consensus_queue.cc:237] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:49.671204 29471 raft_consensus.cc:1261] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:49.671204 32255 raft_consensus.cc:1261] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:49.671626 6411 consensus_queue.cc:1035] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:49.671797 7191 consensus_queue.cc:1035] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:49.673964 7431 raft_consensus.cc:2932] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:49.674273 32255 raft_consensus.cc:2932] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:49.674374 29471 raft_consensus.cc:2932] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:49.677357 29323 catalog_manager.cc:5484] T b19374a33aaf41069e98fd035d2b9f00 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:49.680819 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet b19374a33aaf41069e98fd035d2b9f00 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:49.681205 7433 tablet_replica.cc:330] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:49.682034 7433 raft_consensus.cc:2229] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:49.682245 7433 raft_consensus.cc:2258] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.682729 7433 ts_tablet_manager.cc:1875] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:49.685171 7433 ts_tablet_manager.cc:1888] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:49.685318 7433 log.cc:1196] T b19374a33aaf41069e98fd035d2b9f00 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/b19374a33aaf41069e98fd035d2b9f00
I20240502 14:00:49.686400 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet b19374a33aaf41069e98fd035d2b9f00 (table rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]) successfully deleted
I20240502 14:00:49.691052 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:49.699215 7359 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:49.700464 7189 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:49.704912 7360 tablet.cc:1722] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:49.707060 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=ebaee5aa390046d7aba6ada87d472ecd] alter complete (version 3)
I20240502 14:00:49.773007 6975 raft_consensus.cc:482] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:49.773144 6975 raft_consensus.cc:504] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.773547 6975 leader_election.cc:288] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:49.773880 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d11f638cfa4b53b532c9084fa45610" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:49.774044 29729 raft_consensus.cc:2454] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:49.774423 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d11f638cfa4b53b532c9084fa45610" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:49.774549 31859 raft_consensus.cc:2454] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:49.774962 31926 leader_election.cc:302] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:49.775620 6975 raft_consensus.cc:2781] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:49.775686 6975 raft_consensus.cc:482] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:49.775725 6975 raft_consensus.cc:3037] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.776852 6975 raft_consensus.cc:504] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.777217 6975 leader_election.cc:288] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:49.778023 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d11f638cfa4b53b532c9084fa45610" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:49.778158 31859 raft_consensus.cc:3037] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.779174 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d11f638cfa4b53b532c9084fa45610" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:49.779304 29729 raft_consensus.cc:3037] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.780265 29729 raft_consensus.cc:2454] T 00d11f638cfa4b53b532c9084fa45610 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:49.781512 31926 leader_election.cc:302] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:49.782152 6975 raft_consensus.cc:2781] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:49.782370 6975 raft_consensus.cc:686] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:49.782413 31859 raft_consensus.cc:2454] T 00d11f638cfa4b53b532c9084fa45610 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:49.782532 6975 consensus_queue.cc:237] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:49.783711 7479 raft_consensus.cc:482] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:49.783811 7479 raft_consensus.cc:504] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.786450 29324 catalog_manager.cc:5484] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:49.788718 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad144c461044c299ea905832908666a" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:49.788857 29729 raft_consensus.cc:2454] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:49.789170 29405 leader_election.cc:302] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:49.789714 6543 raft_consensus.cc:2781] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:49.789785 6543 raft_consensus.cc:482] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:49.789866 6543 raft_consensus.cc:3037] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.790282 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad144c461044c299ea905832908666a" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:49.790412 31859 raft_consensus.cc:2454] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:49.790555 7479 leader_election.cc:288] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:49.791030 6543 raft_consensus.cc:504] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.791394 6543 leader_election.cc:288] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:49.791806 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad144c461044c299ea905832908666a" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:49.791939 29729 raft_consensus.cc:3037] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.792135 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad144c461044c299ea905832908666a" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:49.792227 31859 raft_consensus.cc:3037] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.792923 29729 raft_consensus.cc:2454] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:49.793237 31859 raft_consensus.cc:2454] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:49.793498 29405 leader_election.cc:302] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:49.795233 6543 raft_consensus.cc:2781] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:49.795347 6543 raft_consensus.cc:686] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:49.795490 6543 consensus_queue.cc:237] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.798954 29323 catalog_manager.cc:5484] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:49.809232 29325 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:49.809409 29325 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:49.813169 29325 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 65c13043f597449cbe533eca8c44a060
I20240502 14:00:49.814426 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c13043f597449cbe533eca8c44a060 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:49 UTC) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:49.814761 7481 tablet_replica.cc:330] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:49.814983 32234 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c13043f597449cbe533eca8c44a060 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:49 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:49.815272 7482 tablet_replica.cc:330] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:49.815470 7482 raft_consensus.cc:2229] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:49.815599 7482 raft_consensus.cc:2258] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.815639 7481 raft_consensus.cc:2229] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:49.815785 7481 raft_consensus.cc:2258] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.816169 7482 ts_tablet_manager.cc:1875] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:49.816414 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c13043f597449cbe533eca8c44a060 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:49 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:49.817829 7483 tablet_replica.cc:330] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:49.817946 7481 ts_tablet_manager.cc:1875] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:49.818312 7483 raft_consensus.cc:2229] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:49.819020 7482 ts_tablet_manager.cc:1888] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:49.819170 7482 log.cc:1196] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/65c13043f597449cbe533eca8c44a060
I20240502 14:00:49.819331 7483 raft_consensus.cc:2258] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.819530 7482 ts_tablet_manager.cc:1909] T 65c13043f597449cbe533eca8c44a060 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:49.819921 7483 ts_tablet_manager.cc:1875] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:49.820322 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 65c13043f597449cbe533eca8c44a060 (table rebalancer_dynamic_table_new_name [id=ebaee5aa390046d7aba6ada87d472ecd]) successfully deleted
I20240502 14:00:49.820627 7481 ts_tablet_manager.cc:1888] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:49.820705 7481 log.cc:1196] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/65c13043f597449cbe533eca8c44a060
I20240502 14:00:49.821056 7481 ts_tablet_manager.cc:1909] T 65c13043f597449cbe533eca8c44a060 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting consensus metadata
I20240502 14:00:49.821971 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 65c13043f597449cbe533eca8c44a060 (table rebalancer_dynamic_table_new_name [id=ebaee5aa390046d7aba6ada87d472ecd]) successfully deleted
I20240502 14:00:49.823017 7483 ts_tablet_manager.cc:1888] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:49.823099 7483 log.cc:1196] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/65c13043f597449cbe533eca8c44a060
I20240502 14:00:49.823379 7483 ts_tablet_manager.cc:1909] T 65c13043f597449cbe533eca8c44a060 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:49.823948 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 65c13043f597449cbe533eca8c44a060 (table rebalancer_dynamic_table_new_name [id=ebaee5aa390046d7aba6ada87d472ecd]) successfully deleted
I20240502 14:00:49.917955 29325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:49.920656 7431 raft_consensus.cc:482] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:49.920800 7431 raft_consensus.cc:504] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.921326 7431 leader_election.cc:288] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:49.921679 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e9f5068aa48b5a87841bdf720d9e0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:49.921712 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e9f5068aa48b5a87841bdf720d9e0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:49.921833 29471 raft_consensus.cc:2454] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:49.921876 29729 raft_consensus.cc:2454] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:49.922188 29537 leader_election.cc:302] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:49.922417 7431 raft_consensus.cc:2781] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:49.922479 7431 raft_consensus.cc:482] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:49.922530 7431 raft_consensus.cc:3037] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.923574 7431 raft_consensus.cc:504] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.924026 32101 tablet_service.cc:1467] Processing CreateTablet for tablet c735def637154813a7fbeb0931d4796f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]), partition=
I20240502 14:00:49.924299 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e9f5068aa48b5a87841bdf720d9e0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:49.924353 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c735def637154813a7fbeb0931d4796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.924381 29451 tablet_service.cc:1467] Processing CreateTablet for tablet c735def637154813a7fbeb0931d4796f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]), partition=
I20240502 14:00:49.924400 29471 raft_consensus.cc:3037] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.924602 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c735def637154813a7fbeb0931d4796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.925192 29471 raft_consensus.cc:2454] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:49.925762 29537 leader_election.cc:302] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:49.925913 6411 raft_consensus.cc:2781] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:49.925979 6411 raft_consensus.cc:686] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:49.926126 6411 consensus_queue.cc:237] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:49.926607 7401 tablet_bootstrap.cc:492] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:49.926977 31972 tablet_service.cc:1467] Processing CreateTablet for tablet c735def637154813a7fbeb0931d4796f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]), partition=
I20240502 14:00:49.927091 7431 leader_election.cc:288] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:49.927245 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c735def637154813a7fbeb0931d4796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:49.927335 7484 tablet_bootstrap.cc:492] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:49.927632 7401 tablet_bootstrap.cc:654] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.928206 7484 tablet_bootstrap.cc:654] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.928158 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e9f5068aa48b5a87841bdf720d9e0" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:49.928288 29729 raft_consensus.cc:3037] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:49.929339 29729 raft_consensus.cc:2454] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:49.929425 7400 tablet_bootstrap.cc:492] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:49.930490 29325 catalog_manager.cc:5484] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:49.930789 7400 tablet_bootstrap.cc:654] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:49.931785 7401 tablet_bootstrap.cc:492] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:49.931887 7401 ts_tablet_manager.cc:1367] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:49.932088 7484 tablet_bootstrap.cc:492] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:49.932283 7484 ts_tablet_manager.cc:1367] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:49.932354 7401 raft_consensus.cc:348] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.932503 7401 raft_consensus.cc:374] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.932543 7401 raft_consensus.cc:729] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.932679 7401 consensus_queue.cc:260] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.932963 7401 ts_tablet_manager.cc:1398] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:49.932829 7484 raft_consensus.cc:348] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.933775 7484 raft_consensus.cc:374] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.933840 7484 raft_consensus.cc:729] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.934000 7484 consensus_queue.cc:260] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.934345 7484 ts_tablet_manager.cc:1398] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:49.938598 7400 tablet_bootstrap.cc:492] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:49.938712 7400 ts_tablet_manager.cc:1367] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:49.939327 7400 raft_consensus.cc:348] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.939445 7400 raft_consensus.cc:374] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:49.939478 7400 raft_consensus.cc:729] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:49.939596 7400 consensus_queue.cc:260] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:49.939915 7400 ts_tablet_manager.cc:1398] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:49.951784 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:49.954452 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:49.958117 31972 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:49.977630 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:49.981267 31840 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:49.990669 32234 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:49.995522 29712 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:50.017253 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:50.063078 5217 raft_consensus.cc:482] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:50.063203 5217 raft_consensus.cc:504] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:50.063673 5217 leader_election.cc:288] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:50.063897 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c735def637154813a7fbeb0931d4796f" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:50.064023 31992 raft_consensus.cc:2454] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:50.064105 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c735def637154813a7fbeb0931d4796f" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:50.064260 29471 raft_consensus.cc:2454] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:50.064311 32056 leader_election.cc:302] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:50.064765 7237 raft_consensus.cc:2781] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:50.064831 7237 raft_consensus.cc:482] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:50.064864 7237 raft_consensus.cc:3037] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.065948 7237 raft_consensus.cc:504] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:50.066296 7237 leader_election.cc:288] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:50.067023 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c735def637154813a7fbeb0931d4796f" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:50.067147 29471 raft_consensus.cc:3037] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.067899 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c735def637154813a7fbeb0931d4796f" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:50.068009 31992 raft_consensus.cc:3037] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.068089 29471 raft_consensus.cc:2454] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:50.068358 32058 leader_election.cc:302] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:50.068580 5217 raft_consensus.cc:2781] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:50.068666 5217 raft_consensus.cc:686] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:50.068843 5217 consensus_queue.cc:237] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:50.069072 31992 raft_consensus.cc:2454] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:50.071285 29323 catalog_manager.cc:5484] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:50.077489 29601 consensus_queue.cc:237] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:50.078811 32122 raft_consensus.cc:1261] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:50.078811 29863 raft_consensus.cc:1261] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:50.079221 6411 consensus_queue.cc:1035] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:50.079418 7431 consensus_queue.cc:1035] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:50.081102 7431 raft_consensus.cc:2932] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.081518 32119 raft_consensus.cc:2932] T bd91ed1ae1354a84a0ff668efdbc03d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.081725 29863 raft_consensus.cc:2932] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.085086 29323 catalog_manager.cc:5484] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.088315 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet bd91ed1ae1354a84a0ff668efdbc03d9 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:50.088649 7523 tablet_replica.cc:330] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:50.088784 7523 raft_consensus.cc:2229] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:50.088956 7523 raft_consensus.cc:2258] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:50.089458 7523 ts_tablet_manager.cc:1875] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:50.091966 7523 ts_tablet_manager.cc:1888] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:50.092039 7523 log.cc:1196] T bd91ed1ae1354a84a0ff668efdbc03d9 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd91ed1ae1354a84a0ff668efdbc03d9
I20240502 14:00:50.093024 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet bd91ed1ae1354a84a0ff668efdbc03d9 (table rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]) successfully deleted
I20240502 14:00:50.124181 29863 consensus_queue.cc:237] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:50.125720 29601 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:50.125857 29729 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:50.126253 6950 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:50.126536 29797 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be8afc0891fb43fa9ffc863ad2bc90d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:50.126580 6950 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:50.133953 7333 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.134402 29729 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.136984 29601 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.137814 29320 catalog_manager.cc:5484] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:50.173087 6975 consensus_queue.cc:1035] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.176776 7477 consensus_queue.cc:1035] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.195387 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:50.202517 29471 raft_consensus.cc:1261] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:50.202636 31992 raft_consensus.cc:1261] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:50.203313 7237 consensus_queue.cc:1035] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.203581 5217 consensus_queue.cc:1035] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.206403 7604 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:50.207499 29324 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet c735def637154813a7fbeb0931d4796f (table rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]). Expected version 1 got 0
I20240502 14:00:50.209022 29324 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet c735def637154813a7fbeb0931d4796f (table rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]). Expected version 1 got 0
I20240502 14:00:50.210402 7604 tablet.cc:1620] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:50.212217 29328 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet c735def637154813a7fbeb0931d4796f (table rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]). Expected version 1 got 0
I20240502 14:00:50.215562 7605 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:50.215932 7605 tablet.cc:1620] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:50.219616 29323 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet c735def637154813a7fbeb0931d4796f (table rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]). Expected version 1 got 0
I20240502 14:00:50.224754 7606 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:50.225363 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea] alter complete (version 1)
I20240502 14:00:50.229413 7606 tablet.cc:1620] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:50.279793 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:50.280176 31972 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:50.282189 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:50.290988 32101 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:50.296092 32234 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:50.299010 29843 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:50.320025 6543 consensus_queue.cc:1035] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.329324 29712 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:50.334875 31840 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:50.335146 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:50.337256 6543 consensus_queue.cc:1035] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.351326 7604 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:50.351680 7604 tablet.cc:1620] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:50.353030 7605 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:50.353344 7605 tablet.cc:1620] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:50.360118 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea] alter complete (version 2)
I20240502 14:00:50.360262 6411 consensus_queue.cc:1035] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.367215 7606 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:50.367539 7606 tablet.cc:1620] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:50.374423 7191 consensus_queue.cc:1035] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.466460 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:50.473416 7604 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:50.474145 7606 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:50.475215 7605 tablet.cc:1722] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:50.481489 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:50.488817 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea] alter complete (version 3)
I20240502 14:00:50.490875 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 3e309595cfaf4bf78e6fe9594018696d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.491202 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e309595cfaf4bf78e6fe9594018696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.493671 31840 tablet_service.cc:1467] Processing CreateTablet for tablet a0001d3f124c42eba6129836209a41fa (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.493947 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0001d3f124c42eba6129836209a41fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.495157 31972 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7b45e60d140d4a6e937307370fdc9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.495450 31972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7b45e60d140d4a6e937307370fdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.496925 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7b45e60d140d4a6e937307370fdc9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.497195 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7b45e60d140d4a6e937307370fdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.497949 31971 tablet_service.cc:1467] Processing CreateTablet for tablet a0001d3f124c42eba6129836209a41fa (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.498176 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0001d3f124c42eba6129836209a41fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.500411 31970 tablet_service.cc:1467] Processing CreateTablet for tablet 3e309595cfaf4bf78e6fe9594018696d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.500659 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e309595cfaf4bf78e6fe9594018696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.503571 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7b45e60d140d4a6e937307370fdc9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.503647 29842 tablet_service.cc:1467] Processing CreateTablet for tablet a0001d3f124c42eba6129836209a41fa (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.503834 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7b45e60d140d4a6e937307370fdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.503996 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0001d3f124c42eba6129836209a41fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.504155 7669 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:50.505182 7669 tablet_bootstrap.cc:654] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.508292 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 3e309595cfaf4bf78e6fe9594018696d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:50.508553 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e309595cfaf4bf78e6fe9594018696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.516899 7673 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:50.517946 7673 tablet_bootstrap.cc:654] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.519158 7669 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:50.519320 7669 ts_tablet_manager.cc:1367] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:00:50.521123 7673 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:50.521297 7673 ts_tablet_manager.cc:1367] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:50.522352 7669 raft_consensus.cc:348] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.522619 7669 raft_consensus.cc:374] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.522707 7669 raft_consensus.cc:729] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.522922 7673 raft_consensus.cc:348] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.522959 7669 consensus_queue.cc:260] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.523152 7673 raft_consensus.cc:374] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.523213 7673 raft_consensus.cc:729] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.523351 7670 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:50.523366 7673 consensus_queue.cc:260] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.523936 7668 tablet_bootstrap.cc:492] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:50.524206 7671 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:50.525043 7669 ts_tablet_manager.cc:1398] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:50.525316 7669 tablet_bootstrap.cc:492] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:50.526446 7669 tablet_bootstrap.cc:654] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.528043 7668 tablet_bootstrap.cc:654] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.529644 7670 tablet_bootstrap.cc:654] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.530433 7673 ts_tablet_manager.cc:1398] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:00:50.530789 7671 tablet_bootstrap.cc:654] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.531330 7673 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:50.532752 7671 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:50.532907 7671 ts_tablet_manager.cc:1367] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:50.533157 7668 tablet_bootstrap.cc:492] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:50.533507 7671 raft_consensus.cc:348] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.533690 7671 raft_consensus.cc:374] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.533771 7671 raft_consensus.cc:729] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.534015 7671 consensus_queue.cc:260] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.534373 7671 ts_tablet_manager.cc:1398] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:50.533738 7668 ts_tablet_manager.cc:1367] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:50.535279 7668 raft_consensus.cc:348] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.536717 7668 raft_consensus.cc:374] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.536820 7668 raft_consensus.cc:729] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.537019 7668 consensus_queue.cc:260] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.537592 7670 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:50.537742 7670 ts_tablet_manager.cc:1367] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:00:50.537488 7673 tablet_bootstrap.cc:654] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.538311 7670 raft_consensus.cc:348] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.538506 7670 raft_consensus.cc:374] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.539430 7670 raft_consensus.cc:729] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.540288 7668 ts_tablet_manager.cc:1398] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:50.540256 7670 consensus_queue.cc:260] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.541069 7670 ts_tablet_manager.cc:1398] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:50.542243 7670 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:50.543488 7670 tablet_bootstrap.cc:654] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.543658 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:50.546978 7551 raft_consensus.cc:482] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:50.547080 7551 raft_consensus.cc:504] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.547504 7551 leader_election.cc:288] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:50.547704 7690 ts_tablet_manager.cc:902] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:50.548040 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0001d3f124c42eba6129836209a41fa" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:50.548185 31860 raft_consensus.cc:2454] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:50.548264 7690 tablet_copy_client.cc:308] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:50.548632 29797 leader_election.cc:302] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:50.548626 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0001d3f124c42eba6129836209a41fa" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:50.548791 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet be8afc0891fb43fa9ffc863ad2bc90d2 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:37167)
I20240502 14:00:50.548880 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet be8afc0891fb43fa9ffc863ad2bc90d2 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:37167: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-be8afc0891fb43fa9ffc863ad2bc90d2
I20240502 14:00:50.549947 29884 tablet_copy_source_session.cc:205] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 14:00:50.550046 29797 leader_election.cc:341] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:50.550554 7551 raft_consensus.cc:2781] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:50.550609 7551 raft_consensus.cc:482] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:50.550644 7551 raft_consensus.cc:3037] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.551509 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8afc0891fb43fa9ffc863ad2bc90d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.551817 7551 raft_consensus.cc:504] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.552176 7551 leader_election.cc:288] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:50.553095 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0001d3f124c42eba6129836209a41fa" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:50.553205 31860 raft_consensus.cc:3037] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.553723 31970 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:50.554199 31860 raft_consensus.cc:2454] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:50.554755 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0001d3f124c42eba6129836209a41fa" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:50.554831 29797 leader_election.cc:302] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:50.555083 7551 raft_consensus.cc:2781] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:50.555150 7551 raft_consensus.cc:686] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:50.555264 7551 consensus_queue.cc:237] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
W20240502 14:00:50.555959 29797 leader_election.cc:341] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:50.557363 29322 catalog_manager.cc:5484] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:50.558552 29712 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:50.568928 7670 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:50.569114 7670 ts_tablet_manager.cc:1367] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240502 14:00:50.569661 7670 raft_consensus.cc:348] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.570129 7670 raft_consensus.cc:374] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.570225 7670 raft_consensus.cc:729] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.570238 7673 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:50.570446 7673 ts_tablet_manager.cc:1367] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240502 14:00:50.570446 7670 consensus_queue.cc:260] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.572499 7673 raft_consensus.cc:348] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.572604 7673 raft_consensus.cc:374] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.572631 7673 raft_consensus.cc:729] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.572736 7673 consensus_queue.cc:260] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.573978 7673 ts_tablet_manager.cc:1398] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:50.574674 7670 ts_tablet_manager.cc:1398] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:50.574921 7670 tablet_bootstrap.cc:492] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:50.576735 7670 tablet_bootstrap.cc:654] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.581617 29843 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:50.585179 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:50.586366 32101 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:50.587925 32234 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:50.591010 7690 tablet_copy_client.cc:783] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:50.591195 7690 tablet_copy_client.cc:647] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:50.601577 7690 tablet_copy_client.cc:523] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:50.602965 7669 tablet_bootstrap.cc:492] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:50.603083 7669 ts_tablet_manager.cc:1367] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.002s
I20240502 14:00:50.603574 7669 raft_consensus.cc:348] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.603811 7690 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:50.603922 7669 raft_consensus.cc:374] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.603986 7669 raft_consensus.cc:729] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.604126 7669 consensus_queue.cc:260] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.604434 7669 ts_tablet_manager.cc:1398] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:50.605531 7670 tablet_bootstrap.cc:492] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:50.605620 7670 ts_tablet_manager.cc:1367] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240502 14:00:50.606025 7670 raft_consensus.cc:348] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.606129 7670 raft_consensus.cc:374] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.606163 7670 raft_consensus.cc:729] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.606271 7670 consensus_queue.cc:260] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.606540 7670 ts_tablet_manager.cc:1398] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:50.609334 29321 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:50.609494 29321 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:50.612901 29321 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet c735def637154813a7fbeb0931d4796f
I20240502 14:00:50.612946 31840 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:50.615449 31970 tablet_service.cc:1514] Processing DeleteTablet for tablet c735def637154813a7fbeb0931d4796f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:50 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:50.615427 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet c735def637154813a7fbeb0931d4796f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:50 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:50.619963 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet c735def637154813a7fbeb0931d4796f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:50 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:50.623008 7702 tablet_replica.cc:330] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:50.623203 7702 raft_consensus.cc:2229] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:50.623589 7702 raft_consensus.cc:2258] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:50.624286 7702 ts_tablet_manager.cc:1875] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:50.625625 7700 tablet_replica.cc:330] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:50.625844 7700 raft_consensus.cc:2229] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:50.626061 7700 raft_consensus.cc:2258] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:50.626696 7702 ts_tablet_manager.cc:1888] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:50.626814 7702 log.cc:1196] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/c735def637154813a7fbeb0931d4796f
I20240502 14:00:50.627166 7702 ts_tablet_manager.cc:1909] T c735def637154813a7fbeb0931d4796f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:50.627921 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet c735def637154813a7fbeb0931d4796f (table rebalancer_dynamic_table_new_name [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]) successfully deleted
I20240502 14:00:50.628140 7700 ts_tablet_manager.cc:1875] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:50.629072 7701 tablet_replica.cc:330] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:50.629283 7701 raft_consensus.cc:2229] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:50.630175 7701 raft_consensus.cc:2258] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:50.631544 7701 ts_tablet_manager.cc:1875] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:50.634765 7700 ts_tablet_manager.cc:1888] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:50.636189 7700 log.cc:1196] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/c735def637154813a7fbeb0931d4796f
I20240502 14:00:50.636559 7700 ts_tablet_manager.cc:1909] T c735def637154813a7fbeb0931d4796f P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:50.637298 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet c735def637154813a7fbeb0931d4796f (table rebalancer_dynamic_table_new_name [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]) successfully deleted
I20240502 14:00:50.643610 7701 ts_tablet_manager.cc:1888] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:50.643788 7701 log.cc:1196] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/c735def637154813a7fbeb0931d4796f
I20240502 14:00:50.644234 7701 ts_tablet_manager.cc:1909] T c735def637154813a7fbeb0931d4796f P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:50.645030 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet c735def637154813a7fbeb0931d4796f (table rebalancer_dynamic_table_new_name [id=8b0ddb5a49bd4c249b1dd30d7eb1c6ea]) successfully deleted
I20240502 14:00:50.649144 7690 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:50.649520 7690 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:50.649752 7690 ts_tablet_manager.cc:1367] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.046s user 0.027s sys 0.000s
I20240502 14:00:50.650312 7690 raft_consensus.cc:348] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:50.650532 7690 raft_consensus.cc:729] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.650724 7690 consensus_queue.cc:260] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:50.651162 7690 ts_tablet_manager.cc:1398] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:50.652038 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-be8afc0891fb43fa9ffc863ad2bc90d2 received from {username='slave'} at 127.22.55.135:37167
I20240502 14:00:50.652145 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-be8afc0891fb43fa9ffc863ad2bc90d2 on tablet be8afc0891fb43fa9ffc863ad2bc90d2 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:50.676084 6975 raft_consensus.cc:482] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:50.676203 6975 raft_consensus.cc:504] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.676581 6975 leader_election.cc:288] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:50.676968 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7b45e60d140d4a6e937307370fdc9" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:50.676985 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7b45e60d140d4a6e937307370fdc9" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:50.677124 29732 raft_consensus.cc:2454] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:50.677135 29863 raft_consensus.cc:2454] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:50.677534 31926 leader_election.cc:302] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:50.677695 6975 raft_consensus.cc:2781] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:50.677750 6975 raft_consensus.cc:482] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:50.677799 6975 raft_consensus.cc:3037] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.677874 31990 consensus_queue.cc:237] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:50.678768 6975 raft_consensus.cc:504] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.679126 6975 leader_election.cc:288] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:50.679149 29732 raft_consensus.cc:1261] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:50.679173 29470 raft_consensus.cc:1261] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:50.679587 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7b45e60d140d4a6e937307370fdc9" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:50.679716 29863 raft_consensus.cc:3037] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.679991 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7b45e60d140d4a6e937307370fdc9" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:50.680096 29732 raft_consensus.cc:3037] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.680791 29863 raft_consensus.cc:2454] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:50.680821 7719 consensus_queue.cc:1035] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:50.681133 29732 raft_consensus.cc:2454] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:50.681221 31926 leader_election.cc:302] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:50.681404 7719 raft_consensus.cc:2781] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:50.681480 7719 raft_consensus.cc:686] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:50.681571 7477 consensus_queue.cc:1035] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:50.681636 7719 consensus_queue.cc:237] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.683353 6975 raft_consensus.cc:2932] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:50.684087 29321 catalog_manager.cc:5484] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:50.687502 29470 raft_consensus.cc:2932] T 5edd4906c0914f6c83dc2bb1501d15bc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:50.689733 29321 catalog_manager.cc:5484] T 5edd4906c0914f6c83dc2bb1501d15bc P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:50.692220 32234 tablet_service.cc:1514] Processing DeleteTablet for tablet 5edd4906c0914f6c83dc2bb1501d15bc with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:50.692653 7727 tablet_replica.cc:330] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:50.692643 7719 raft_consensus.cc:482] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:50.692723 7719 raft_consensus.cc:504] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:50.692873 7727 raft_consensus.cc:2229] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:50.693006 7727 raft_consensus.cc:2258] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:50.693092 7719 leader_election.cc:288] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:50.693519 7727 ts_tablet_manager.cc:1875] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:50.693603 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0001d3f124c42eba6129836209a41fa" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:50.693806 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0001d3f124c42eba6129836209a41fa" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:50.693971 31860 raft_consensus.cc:2379] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 269b366a09ff435ea8ec8dc01d0f9f6e in current term 1: Already voted for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in this term.
I20240502 14:00:50.694237 31925 leader_election.cc:302] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:50.694380 7719 raft_consensus.cc:3037] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.695277 7719 raft_consensus.cc:2726] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:00:50.696200 7727 ts_tablet_manager.cc:1888] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:50.696312 7727 log.cc:1196] T 5edd4906c0914f6c83dc2bb1501d15bc P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/5edd4906c0914f6c83dc2bb1501d15bc
I20240502 14:00:50.696808 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 5edd4906c0914f6c83dc2bb1501d15bc (table rebalancer_extra_table_10 [id=79b91ad43b634662a251e1a8f6b0724a]) successfully deleted
I20240502 14:00:50.704025 29732 raft_consensus.cc:2932] T 5edd4906c0914f6c83dc2bb1501d15bc P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:50.717025 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:50.726075 32234 tablet_service.cc:1467] Processing CreateTablet for tablet 8f62b18be6e34ea29c906f71890e7c70 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3]), partition=
I20240502 14:00:50.726404 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f62b18be6e34ea29c906f71890e7c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.728555 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 8f62b18be6e34ea29c906f71890e7c70 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3]), partition=
I20240502 14:00:50.728842 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f62b18be6e34ea29c906f71890e7c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.729141 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 8f62b18be6e34ea29c906f71890e7c70 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3]), partition=
I20240502 14:00:50.729400 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f62b18be6e34ea29c906f71890e7c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:50.731740 7737 tablet_bootstrap.cc:492] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:50.734418 7736 tablet_bootstrap.cc:492] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:50.734561 7737 tablet_bootstrap.cc:654] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.735780 7736 tablet_bootstrap.cc:654] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.737015 7737 tablet_bootstrap.cc:492] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:50.737208 7737 ts_tablet_manager.cc:1367] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:50.737756 7737 raft_consensus.cc:348] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.737946 7737 raft_consensus.cc:374] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.738018 7737 raft_consensus.cc:729] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.738268 7737 consensus_queue.cc:260] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.738813 7735 tablet_bootstrap.cc:492] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:50.739610 7737 ts_tablet_manager.cc:1398] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:50.740164 7736 tablet_bootstrap.cc:492] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:50.740257 7736 ts_tablet_manager.cc:1367] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:50.740689 7736 raft_consensus.cc:348] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.740790 7736 raft_consensus.cc:374] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.740818 7736 raft_consensus.cc:729] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.740887 7735 tablet_bootstrap.cc:654] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:50.740918 7736 consensus_queue.cc:260] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.741164 7736 ts_tablet_manager.cc:1398] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:50.754776 7735 tablet_bootstrap.cc:492] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:50.755012 7735 ts_tablet_manager.cc:1367] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:00:50.755532 7735 raft_consensus.cc:348] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.755645 7735 raft_consensus.cc:374] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:50.755677 7735 raft_consensus.cc:729] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:50.755793 7735 consensus_queue.cc:260] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.756119 7735 ts_tablet_manager.cc:1398] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:50.803016 6618 raft_consensus.cc:482] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:50.803144 6618 raft_consensus.cc:504] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.803553 6618 leader_election.cc:288] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:50.803902 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f62b18be6e34ea29c906f71890e7c70" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:50.804039 29601 raft_consensus.cc:2454] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:50.804293 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f62b18be6e34ea29c906f71890e7c70" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:50.804391 32119 raft_consensus.cc:2454] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 0.
I20240502 14:00:50.804381 32188 leader_election.cc:302] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:50.804764 6618 raft_consensus.cc:2781] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:50.804828 6618 raft_consensus.cc:482] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:50.804868 6618 raft_consensus.cc:3037] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.806017 6618 raft_consensus.cc:504] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.806296 6618 leader_election.cc:288] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:50.806815 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f62b18be6e34ea29c906f71890e7c70" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:50.806947 29601 raft_consensus.cc:3037] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.808002 29601 raft_consensus.cc:2454] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:50.808676 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f62b18be6e34ea29c906f71890e7c70" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:50.808799 32119 raft_consensus.cc:3037] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.809845 32119 raft_consensus.cc:2454] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:00:50.810398 32188 leader_election.cc:302] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:50.810681 6618 raft_consensus.cc:2781] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:50.810760 6618 raft_consensus.cc:686] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:00:50.811249 6618 consensus_queue.cc:237] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:50.813936 29320 catalog_manager.cc:5484] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:50.881378 7719 raft_consensus.cc:482] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:50.881532 7719 raft_consensus.cc:504] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.881943 7719 leader_election.cc:288] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:50.882329 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e309595cfaf4bf78e6fe9594018696d" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:50.882329 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e309595cfaf4bf78e6fe9594018696d" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:50.882478 31860 raft_consensus.cc:2454] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:50.882486 29470 raft_consensus.cc:2454] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:50.882778 31925 leader_election.cc:302] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:50.882987 7719 raft_consensus.cc:2781] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:50.883072 7719 raft_consensus.cc:482] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:50.883127 7719 raft_consensus.cc:3037] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.884095 7719 raft_consensus.cc:504] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.884430 7719 leader_election.cc:288] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:50.884733 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e309595cfaf4bf78e6fe9594018696d" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:50.884742 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e309595cfaf4bf78e6fe9594018696d" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:50.884855 31860 raft_consensus.cc:3037] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.884884 29470 raft_consensus.cc:3037] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:50.885893 31860 raft_consensus.cc:2454] T 3e309595cfaf4bf78e6fe9594018696d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:50.885950 29470 raft_consensus.cc:2454] T 3e309595cfaf4bf78e6fe9594018696d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:50.886272 31927 leader_election.cc:302] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:50.886469 7719 raft_consensus.cc:2781] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:50.886557 7719 raft_consensus.cc:686] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:50.886699 7719 consensus_queue.cc:237] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:50.889672 29320 catalog_manager.cc:5484] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:50.912439 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:50.917129 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:50.918362 29712 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:50.923821 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:50.930697 29843 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:50.936393 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:50.937765 7551 consensus_queue.cc:1035] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.940171 31840 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:50.946955 6950 consensus_queue.cc:1035] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.951131 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:50.967103 32234 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:50.973413 29600 raft_consensus.cc:1261] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:50.973846 6618 consensus_queue.cc:1035] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.979876 29320 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 8f62b18be6e34ea29c906f71890e7c70 (table rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3]). Expected version 1 got 0
I20240502 14:00:50.983522 32119 raft_consensus.cc:1261] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:50.983942 6618 consensus_queue.cc:1035] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:50.987865 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 8f62b18be6e34ea29c906f71890e7c70 (table rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3]). Expected version 1 got 0
I20240502 14:00:50.995479 7794 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:50.995826 7794 tablet.cc:1620] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:50.997229 29320 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 8f62b18be6e34ea29c906f71890e7c70 (table rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3]). Expected version 1 got 0
I20240502 14:00:51.002669 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3] alter complete (version 1)
I20240502 14:00:51.004756 7796 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:51.005231 7796 tablet.cc:1620] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:51.006321 7795 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:51.006698 7795 tablet.cc:1620] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:51.039721 29732 consensus_queue.cc:237] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:51.041034 32119 raft_consensus.cc:1261] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:51.041122 29863 raft_consensus.cc:1261] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:51.041496 7622 consensus_queue.cc:1035] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:51.041828 7622 consensus_queue.cc:1035] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:51.043483 32119 raft_consensus.cc:1203] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:51.043416 7622 raft_consensus.cc:2932] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.045511 29863 raft_consensus.cc:2932] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.048733 29323 catalog_manager.cc:5484] T 09ce1f1d33dd4beca86a484a6f498175 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.050266 32119 raft_consensus.cc:2932] T 09ce1f1d33dd4beca86a484a6f498175 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.051455 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ce1f1d33dd4beca86a484a6f498175 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:51.054711 7816 tablet_replica.cc:330] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:51.054994 7816 raft_consensus.cc:2229] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:51.055281 7816 raft_consensus.cc:2258] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:51.055855 7816 ts_tablet_manager.cc:1875] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:51.058035 7816 ts_tablet_manager.cc:1888] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:51.058107 7816 log.cc:1196] T 09ce1f1d33dd4beca86a484a6f498175 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/09ce1f1d33dd4beca86a484a6f498175
I20240502 14:00:51.058698 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 09ce1f1d33dd4beca86a484a6f498175 (table rebalance_test_table_17 [id=959115a316ce4f12bab7b305af4bf7f0]) successfully deleted
I20240502 14:00:51.104578 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:51.111637 7794 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:51.111974 7794 tablet.cc:1620] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:51.112071 7795 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:51.112354 7795 tablet.cc:1620] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:51.113585 7796 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:51.113924 7796 tablet.cc:1620] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:51.117779 29322 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=f1eb2247b2114756a8a5a5ef74dc97d3] alter complete (version 2)
I20240502 14:00:51.156158 7719 consensus_queue.cc:1035] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.159322 7719 consensus_queue.cc:1035] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.191056 29843 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:51.191685 32234 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:51.193908 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:51.200352 31840 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:51.208462 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:51.212039 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:51.217077 29712 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:51.226465 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:51.240172 7794 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:51.243557 7795 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:51.249284 7796 tablet.cc:1722] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:51.249806 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=f1eb2247b2114756a8a5a5ef74dc97d3] alter complete (version 3)
I20240502 14:00:51.252847 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:51.279083 6975 consensus_queue.cc:1035] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.283227 7719 consensus_queue.cc:1035] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.305269 31990 consensus_queue.cc:237] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:51.306486 31859 raft_consensus.cc:1261] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:51.306810 29729 raft_consensus.cc:1261] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:51.306828 6975 consensus_queue.cc:1035] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:51.307389 6975 consensus_queue.cc:1035] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:51.308495 7719 raft_consensus.cc:2932] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:51.308893 31859 raft_consensus.cc:2932] T e574c148957f44bf9c2935e955de99aa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:51.309495 29729 raft_consensus.cc:2932] T e574c148957f44bf9c2935e955de99aa P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:51.312122 29320 catalog_manager.cc:5484] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:51.314823 32234 tablet_service.cc:1514] Processing DeleteTablet for tablet e574c148957f44bf9c2935e955de99aa with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:51.316356 7895 tablet_replica.cc:330] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:51.316548 7895 raft_consensus.cc:2229] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:51.316705 7895 raft_consensus.cc:2258] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:51.317255 7895 ts_tablet_manager.cc:1875] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:51.319540 7895 ts_tablet_manager.cc:1888] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:51.319656 7895 log.cc:1196] T e574c148957f44bf9c2935e955de99aa P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/e574c148957f44bf9c2935e955de99aa
I20240502 14:00:51.320143 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet e574c148957f44bf9c2935e955de99aa (table rebalancer_extra_table_2 [id=8834dcf7510a4edc80f64a8492d25e06]) successfully deleted
I20240502 14:00:51.356539 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:51.356686 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:51.358995 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 8f62b18be6e34ea29c906f71890e7c70
I20240502 14:00:51.359757 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f62b18be6e34ea29c906f71890e7c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:51 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:51.359799 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f62b18be6e34ea29c906f71890e7c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:51 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:51.359798 32234 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f62b18be6e34ea29c906f71890e7c70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:51 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:51.360005 7895 tablet_replica.cc:330] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:51.360149 7895 raft_consensus.cc:2229] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:51.360355 7895 raft_consensus.cc:2258] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:51.360544 7816 tablet_replica.cc:330] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:51.360663 7816 raft_consensus.cc:2229] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:51.360782 7816 raft_consensus.cc:2258] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:51.360896 7895 ts_tablet_manager.cc:1875] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:51.361407 7816 ts_tablet_manager.cc:1875] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:51.363585 7816 ts_tablet_manager.cc:1888] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:51.363588 7895 ts_tablet_manager.cc:1888] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:51.363648 7816 log.cc:1196] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f62b18be6e34ea29c906f71890e7c70
I20240502 14:00:51.363660 7895 log.cc:1196] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/8f62b18be6e34ea29c906f71890e7c70
I20240502 14:00:51.363909 7816 ts_tablet_manager.cc:1909] T 8f62b18be6e34ea29c906f71890e7c70 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:51.364537 7895 ts_tablet_manager.cc:1909] T 8f62b18be6e34ea29c906f71890e7c70 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:51.365525 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 8f62b18be6e34ea29c906f71890e7c70 (table rebalancer_dynamic_table_new_name [id=f1eb2247b2114756a8a5a5ef74dc97d3]) successfully deleted
I20240502 14:00:51.365697 7926 tablet_replica.cc:330] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:51.365898 7926 raft_consensus.cc:2229] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:51.366051 7926 raft_consensus.cc:2258] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:51.366182 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 8f62b18be6e34ea29c906f71890e7c70 (table rebalancer_dynamic_table_new_name [id=f1eb2247b2114756a8a5a5ef74dc97d3]) successfully deleted
I20240502 14:00:51.366595 7926 ts_tablet_manager.cc:1875] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:51.368614 7926 ts_tablet_manager.cc:1888] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:51.368676 7926 log.cc:1196] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/8f62b18be6e34ea29c906f71890e7c70
I20240502 14:00:51.368924 7926 ts_tablet_manager.cc:1909] T 8f62b18be6e34ea29c906f71890e7c70 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:51.369555 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 8f62b18be6e34ea29c906f71890e7c70 (table rebalancer_dynamic_table_new_name [id=f1eb2247b2114756a8a5a5ef74dc97d3]) successfully deleted
I20240502 14:00:51.371157 29729 consensus_queue.cc:237] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.372586 29470 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:51.372717 29668 consensus_peers.cc:485] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92cc4394003840a5b1d59b0fe72c5164. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:51.372778 29600 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:51.373015 7813 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:51.374019 7813 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:51.376044 7813 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.376468 29600 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.377920 29470 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.379429 29323 catalog_manager.cc:5484] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.422925 29322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:51.431401 32234 tablet_service.cc:1467] Processing CreateTablet for tablet 764d8861015246c6a37dfdbd1e3f48e2 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.431731 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764d8861015246c6a37dfdbd1e3f48e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.432011 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 764d8861015246c6a37dfdbd1e3f48e2 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.432147 32233 tablet_service.cc:1467] Processing CreateTablet for tablet bf3e82b3dc394648a13eafe4ec832fb0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.432292 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764d8861015246c6a37dfdbd1e3f48e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.432348 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3e82b3dc394648a13eafe4ec832fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.432448 29580 tablet_service.cc:1467] Processing CreateTablet for tablet bf3e82b3dc394648a13eafe4ec832fb0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.432652 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3e82b3dc394648a13eafe4ec832fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.434230 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 764d8861015246c6a37dfdbd1e3f48e2 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.434522 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764d8861015246c6a37dfdbd1e3f48e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.434713 29842 tablet_service.cc:1467] Processing CreateTablet for tablet 631725c869674c45ac79c3f05cee8356 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.434950 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 631725c869674c45ac79c3f05cee8356 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.435127 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631725c869674c45ac79c3f05cee8356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.435216 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631725c869674c45ac79c3f05cee8356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.436949 29843 tablet_service.cc:1467] Processing CreateTablet for tablet bf3e82b3dc394648a13eafe4ec832fb0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.437209 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3e82b3dc394648a13eafe4ec832fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.438598 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 631725c869674c45ac79c3f05cee8356 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:51.438868 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631725c869674c45ac79c3f05cee8356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.439669 7976 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:51.441527 7975 tablet_bootstrap.cc:492] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:51.443818 7973 tablet_bootstrap.cc:492] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:51.444021 7976 tablet_bootstrap.cc:654] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.444789 7973 tablet_bootstrap.cc:654] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.445317 7974 tablet_bootstrap.cc:492] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:51.445608 7975 tablet_bootstrap.cc:654] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.448089 7973 tablet_bootstrap.cc:492] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:51.448184 7973 ts_tablet_manager.cc:1367] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:51.448237 7974 tablet_bootstrap.cc:654] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.448637 7973 raft_consensus.cc:348] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.448746 7973 raft_consensus.cc:374] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.448781 7973 raft_consensus.cc:729] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.448891 7973 consensus_queue.cc:260] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.449182 7973 ts_tablet_manager.cc:1398] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:51.449314 7976 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:51.449431 7976 ts_tablet_manager.cc:1367] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:51.449594 7973 tablet_bootstrap.cc:492] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:51.450183 7976 raft_consensus.cc:348] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.450284 7976 raft_consensus.cc:374] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.450315 7976 raft_consensus.cc:729] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.450467 7973 tablet_bootstrap.cc:654] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.450456 7976 consensus_queue.cc:260] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.450788 7976 ts_tablet_manager.cc:1398] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:51.452297 7974 tablet_bootstrap.cc:492] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:51.452389 7974 ts_tablet_manager.cc:1367] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:51.452790 7974 raft_consensus.cc:348] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.452894 7974 raft_consensus.cc:374] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.452908 7973 tablet_bootstrap.cc:492] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:51.452945 7974 raft_consensus.cc:729] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.453002 7973 ts_tablet_manager.cc:1367] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:51.453079 7974 consensus_queue.cc:260] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.453382 7974 ts_tablet_manager.cc:1398] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:51.453393 7973 raft_consensus.cc:348] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.453492 7973 raft_consensus.cc:374] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.453534 7973 raft_consensus.cc:729] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.453541 7974 tablet_bootstrap.cc:492] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:51.453670 7973 consensus_queue.cc:260] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.453933 7973 ts_tablet_manager.cc:1398] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:51.454093 7973 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:51.454483 7977 tablet_bootstrap.cc:492] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:51.454797 7974 tablet_bootstrap.cc:654] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.454919 7973 tablet_bootstrap.cc:654] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.455571 7977 tablet_bootstrap.cc:654] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.455895 7974 tablet_bootstrap.cc:492] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:51.455974 7974 ts_tablet_manager.cc:1367] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:51.456427 7974 raft_consensus.cc:348] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.456521 7974 raft_consensus.cc:374] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.456557 7974 raft_consensus.cc:729] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.456672 7974 consensus_queue.cc:260] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.456929 7974 ts_tablet_manager.cc:1398] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:51.457360 7973 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:51.457434 7973 ts_tablet_manager.cc:1367] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:51.457860 7973 raft_consensus.cc:348] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.457952 7973 raft_consensus.cc:374] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.457986 7973 raft_consensus.cc:729] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.458107 7973 consensus_queue.cc:260] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.458393 7973 ts_tablet_manager.cc:1398] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:51.460671 7975 tablet_bootstrap.cc:492] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:51.460769 7975 ts_tablet_manager.cc:1367] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:00:51.461234 7975 raft_consensus.cc:348] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.461344 7975 raft_consensus.cc:374] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.461388 7975 raft_consensus.cc:729] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.461355 29321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:51.461512 7975 consensus_queue.cc:260] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.461784 7975 ts_tablet_manager.cc:1398] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:51.463282 7977 tablet_bootstrap.cc:492] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:51.463384 7977 ts_tablet_manager.cc:1367] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:51.466339 7977 raft_consensus.cc:348] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.466481 7977 raft_consensus.cc:374] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.466571 7977 raft_consensus.cc:729] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.466730 7977 consensus_queue.cc:260] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.468077 7977 ts_tablet_manager.cc:1398] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:51.470036 29582 tablet_service.cc:1467] Processing CreateTablet for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083]), partition=
I20240502 14:00:51.470306 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be35d71a8b9f41a59aa8d0d1b5d09c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.470343 32233 tablet_service.cc:1467] Processing CreateTablet for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083]), partition=
I20240502 14:00:51.470546 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be35d71a8b9f41a59aa8d0d1b5d09c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.472762 7974 tablet_bootstrap.cc:492] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:51.473445 7973 tablet_bootstrap.cc:492] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:51.473671 7974 tablet_bootstrap.cc:654] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.474259 7973 tablet_bootstrap.cc:654] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.474694 7974 tablet_bootstrap.cc:492] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:51.474798 7974 ts_tablet_manager.cc:1367] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:51.475054 29451 tablet_service.cc:1467] Processing CreateTablet for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083]), partition=
I20240502 14:00:51.475299 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be35d71a8b9f41a59aa8d0d1b5d09c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.475250 7974 raft_consensus.cc:348] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.475344 7974 raft_consensus.cc:374] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.475394 7974 raft_consensus.cc:729] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.475497 7974 consensus_queue.cc:260] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.475811 7974 ts_tablet_manager.cc:1398] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:51.481196 7977 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:51.482192 7977 tablet_bootstrap.cc:654] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.483341 7977 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:51.483415 7977 ts_tablet_manager.cc:1367] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:51.483786 7977 raft_consensus.cc:348] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.483880 7977 raft_consensus.cc:374] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.483902 7977 raft_consensus.cc:729] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.484012 7977 consensus_queue.cc:260] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.484311 7977 ts_tablet_manager.cc:1398] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:51.484731 7973 tablet_bootstrap.cc:492] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:51.484814 7973 ts_tablet_manager.cc:1367] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240502 14:00:51.485285 7973 raft_consensus.cc:348] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.485386 7973 raft_consensus.cc:374] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.485428 7973 raft_consensus.cc:729] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.485545 7973 consensus_queue.cc:260] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.485864 7973 ts_tablet_manager.cc:1398] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:51.509864 7990 tablet_bootstrap.cc:492] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:51.509948 6614 raft_consensus.cc:482] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:51.510049 6614 raft_consensus.cc:504] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.510455 6614 leader_election.cc:288] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:51.510905 7990 tablet_bootstrap.cc:654] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:51.511242 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631725c869674c45ac79c3f05cee8356" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:51.511346 29863 raft_consensus.cc:2454] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:51.511767 31795 leader_election.cc:302] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:51.512044 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631725c869674c45ac79c3f05cee8356" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:51.512185 32255 raft_consensus.cc:2454] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:51.512706 6614 raft_consensus.cc:2781] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:51.512766 6614 raft_consensus.cc:482] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:51.512821 6614 raft_consensus.cc:3037] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.514008 6614 raft_consensus.cc:504] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.514362 6614 leader_election.cc:288] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:51.515192 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631725c869674c45ac79c3f05cee8356" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:51.515192 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631725c869674c45ac79c3f05cee8356" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:51.515298 29863 raft_consensus.cc:3037] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.515297 32255 raft_consensus.cc:3037] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.516263 29863 raft_consensus.cc:2454] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:51.516263 32255 raft_consensus.cc:2454] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:51.518184 31796 leader_election.cc:302] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:51.518357 6614 raft_consensus.cc:2781] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:51.518543 6614 raft_consensus.cc:686] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:51.521339 6614 consensus_queue.cc:237] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.527257 7990 tablet_bootstrap.cc:492] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:51.528358 29322 catalog_manager.cc:5484] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.530014 7990 ts_tablet_manager.cc:1367] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240502 14:00:51.530536 7990 raft_consensus.cc:348] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.530704 7990 raft_consensus.cc:374] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:51.530776 7990 raft_consensus.cc:729] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.530952 7990 consensus_queue.cc:260] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.531608 7990 ts_tablet_manager.cc:1398] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:51.551046 29451 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:51.554263 29712 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:51.571797 29842 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:51.573958 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:51.574756 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:51.595628 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:51.605475 7789 raft_consensus.cc:482] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:51.605597 7789 raft_consensus.cc:504] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.606050 7789 leader_election.cc:288] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:51.607322 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3e82b3dc394648a13eafe4ec832fb0" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:51.607455 29600 raft_consensus.cc:2454] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:51.607906 29797 leader_election.cc:302] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:51.611303 7789 raft_consensus.cc:2781] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:51.611377 7789 raft_consensus.cc:482] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:51.611413 7789 raft_consensus.cc:3037] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.612622 7789 raft_consensus.cc:504] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.612949 7789 leader_election.cc:288] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:51.613327 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3e82b3dc394648a13eafe4ec832fb0" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:51.613476 29602 raft_consensus.cc:3037] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.614498 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3e82b3dc394648a13eafe4ec832fb0" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:51.614619 29602 raft_consensus.cc:2454] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:51.614634 32255 raft_consensus.cc:2454] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:51.614815 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3e82b3dc394648a13eafe4ec832fb0" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:51.614936 32254 raft_consensus.cc:3037] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.615079 29797 leader_election.cc:302] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:51.615303 7789 raft_consensus.cc:2781] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:51.615375 7789 raft_consensus.cc:686] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:51.615731 7789 consensus_queue.cc:237] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:51.615830 32254 raft_consensus.cc:2454] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:51.618103 29323 catalog_manager.cc:5484] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:51.621230 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:51.622992 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:51.685851 7813 raft_consensus.cc:482] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:51.685961 7813 raft_consensus.cc:504] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.686404 7813 leader_election.cc:288] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:51.687441 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764d8861015246c6a37dfdbd1e3f48e2" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:51.687547 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764d8861015246c6a37dfdbd1e3f48e2" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:51.687602 29602 raft_consensus.cc:2454] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:51.687695 32255 raft_consensus.cc:2454] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:51.687841 29666 leader_election.cc:302] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:51.687979 7813 raft_consensus.cc:2781] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:51.688041 7813 raft_consensus.cc:482] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:51.688091 7813 raft_consensus.cc:3037] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.689216 7813 raft_consensus.cc:504] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.689579 7813 leader_election.cc:288] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:51.690026 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764d8861015246c6a37dfdbd1e3f48e2" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:51.690152 29602 raft_consensus.cc:3037] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.691226 29602 raft_consensus.cc:2454] T 764d8861015246c6a37dfdbd1e3f48e2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:51.691366 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764d8861015246c6a37dfdbd1e3f48e2" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:51.691483 32255 raft_consensus.cc:3037] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.691552 29666 leader_election.cc:302] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:51.691710 7813 raft_consensus.cc:2781] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:51.691788 7813 raft_consensus.cc:686] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:51.691903 7813 consensus_queue.cc:237] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:51.692476 32255 raft_consensus.cc:2454] T 764d8861015246c6a37dfdbd1e3f48e2 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:51.695984 29322 catalog_manager.cc:5484] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:51.790217 7872 raft_consensus.cc:482] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:51.790364 7872 raft_consensus.cc:504] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.790757 7872 leader_election.cc:288] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:51.791119 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be35d71a8b9f41a59aa8d0d1b5d09c83" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:51.791137 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be35d71a8b9f41a59aa8d0d1b5d09c83" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:51.791262 29602 raft_consensus.cc:2454] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:51.791267 32255 raft_consensus.cc:2454] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:51.791554 29404 leader_election.cc:302] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:51.791745 7872 raft_consensus.cc:2781] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:51.791800 7872 raft_consensus.cc:482] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:51.791838 7872 raft_consensus.cc:3037] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.792846 7872 raft_consensus.cc:504] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.793219 7872 leader_election.cc:288] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:51.793697 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be35d71a8b9f41a59aa8d0d1b5d09c83" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:51.793807 29602 raft_consensus.cc:3037] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.793810 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be35d71a8b9f41a59aa8d0d1b5d09c83" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:51.793905 32254 raft_consensus.cc:3037] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:51.794801 32254 raft_consensus.cc:2454] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:51.794801 29602 raft_consensus.cc:2454] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:51.795074 29406 leader_election.cc:302] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:51.795357 7872 raft_consensus.cc:2781] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:51.795457 7872 raft_consensus.cc:686] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:51.795609 7872 consensus_queue.cc:237] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.797925 29329 catalog_manager.cc:5484] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.821138 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:51.824024 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:51.829618 29842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:51.830636 29451 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:51.839919 8061 ts_tablet_manager.cc:902] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:51.840535 8061 tablet_copy_client.cc:308] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:51.841080 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 92cc4394003840a5b1d59b0fe72c5164 from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57195)
I20240502 14:00:51.841212 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 92cc4394003840a5b1d59b0fe72c5164 from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57195: session id = 8973f2527acb48f5ad36589652de73e1-92cc4394003840a5b1d59b0fe72c5164
I20240502 14:00:51.842208 29753 tablet_copy_source_session.cc:205] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:51.846323 29712 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:51.847005 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:51.852528 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:51.855796 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cc4394003840a5b1d59b0fe72c5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:51.858294 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:51.859095 8061 tablet_copy_client.cc:783] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:51.859357 8061 tablet_copy_client.cc:647] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:51.863150 8061 tablet_copy_client.cc:523] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:51.879001 8061 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:51.902043 8061 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:51.902386 8061 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:51.902585 8061 ts_tablet_manager.cc:1367] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.000s
I20240502 14:00:51.903102 8061 raft_consensus.cc:348] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.903550 8061 raft_consensus.cc:729] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:51.903702 8061 consensus_queue.cc:260] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.903837 7235 consensus_queue.cc:1035] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.905508 8061 ts_tablet_manager.cc:1398] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 14:00:51.909039 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-92cc4394003840a5b1d59b0fe72c5164 received from {username='slave'} at 127.22.55.136:57195
I20240502 14:00:51.909134 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-92cc4394003840a5b1d59b0fe72c5164 on tablet 92cc4394003840a5b1d59b0fe72c5164 with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:51.911026 6614 consensus_queue.cc:1035] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.930369 29863 consensus_queue.cc:237] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:51.931524 32122 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:51.931524 29729 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:51.931851 7789 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:51.932049 8067 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:51.933725 7551 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.934284 29729 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.935089 32122 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.938334 29329 catalog_manager.cc:5484] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:51.944626 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet be8afc0891fb43fa9ffc863ad2bc90d2 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:51.945992 8098 tablet_replica.cc:330] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:51.946154 8098 raft_consensus.cc:2229] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:51.946322 8098 raft_consensus.cc:2258] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:51.946879 8098 ts_tablet_manager.cc:1875] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:51.949115 8098 ts_tablet_manager.cc:1888] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:51.949183 8098 log.cc:1196] T be8afc0891fb43fa9ffc863ad2bc90d2 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/be8afc0891fb43fa9ffc863ad2bc90d2
I20240502 14:00:51.949723 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet be8afc0891fb43fa9ffc863ad2bc90d2 (table rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]) successfully deleted
I20240502 14:00:51.968712 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:51.973400 32254 raft_consensus.cc:1261] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:51.973801 8048 consensus_queue.cc:1035] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.973896 29602 raft_consensus.cc:1261] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:51.974300 8048 consensus_queue.cc:1035] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:51.977154 29326 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (table rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083]). Expected version 1 got 0
I20240502 14:00:51.978163 8124 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:51.978160 8123 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:51.978478 8123 tablet.cc:1620] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:51.978636 8124 tablet.cc:1620] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:51.979508 29326 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (table rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083]). Expected version 1 got 0
I20240502 14:00:51.985584 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083] alter complete (version 1)
I20240502 14:00:51.985705 8130 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:51.985981 8130 tablet.cc:1620] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:51.986523 29322 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (table rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083]). Expected version 1 got 0
I20240502 14:00:51.989054 29864 consensus_queue.cc:237] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:51.990876 29470 raft_consensus.cc:1261] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:51.990921 29729 raft_consensus.cc:1261] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:51.991230 29799 consensus_peers.cc:485] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b6ced4716f64a3aaad4cd5299e46d13. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:51.991268 7789 consensus_queue.cc:1035] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:51.991441 7551 consensus_queue.cc:1035] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:51.993429 7789 raft_consensus.cc:2932] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.993889 29729 raft_consensus.cc:2932] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.995613 29470 raft_consensus.cc:2932] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:51.997155 29326 catalog_manager.cc:5484] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.006815 7551 consensus_queue.cc:1035] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.011425 7551 consensus_queue.cc:1035] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.020649 29470 consensus_queue.cc:237] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:52.022037 29729 raft_consensus.cc:1261] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:52.022145 29602 raft_consensus.cc:1261] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:52.022449 29406 consensus_peers.cc:485] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86808085d86746a9bea7fe0ea0647d5f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:52.022500 8152 consensus_queue.cc:1035] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240502 14:00:52.024204 8048 consensus_queue.cc:1035] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:52.027868 8048 raft_consensus.cc:2932] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.028288 29602 raft_consensus.cc:2932] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.030810 29729 raft_consensus.cc:2932] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.031354 29322 catalog_manager.cc:5484] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 4, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.089702 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:52.098701 8123 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:52.099047 8123 tablet.cc:1620] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:52.099738 8130 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:52.100034 8130 tablet.cc:1620] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:52.101025 8124 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:52.101305 8124 tablet.cc:1620] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:52.106945 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=e06e48e5d9ac4f49a28af0ebbc0c3083] alter complete (version 2)
I20240502 14:00:52.181440 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:52.183231 29451 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:52.186020 29842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:52.188186 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.199681 29582 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:52.208236 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:52.215313 32101 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:52.217020 8123 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:52.217911 8124 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:52.219431 8130 tablet.cc:1722] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:52.219789 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:52.223091 32254 raft_consensus.cc:1203] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:52.225529 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=e06e48e5d9ac4f49a28af0ebbc0c3083] alter complete (version 3)
I20240502 14:00:52.232894 29712 tablet_service.cc:1467] Processing CreateTablet for tablet 65229e977b8543639fb27bbbc3b5e797 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.233162 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65229e977b8543639fb27bbbc3b5e797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.234331 31970 tablet_service.cc:1467] Processing CreateTablet for tablet 65229e977b8543639fb27bbbc3b5e797 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.234627 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65229e977b8543639fb27bbbc3b5e797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.235527 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 65229e977b8543639fb27bbbc3b5e797 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.235756 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65229e977b8543639fb27bbbc3b5e797. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.237622 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 444e705b42e7423b9ee1d187ea1de308 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.237767 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 444e705b42e7423b9ee1d187ea1de308 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.237870 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444e705b42e7423b9ee1d187ea1de308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.238020 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444e705b42e7423b9ee1d187ea1de308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.238224 29842 tablet_service.cc:1467] Processing CreateTablet for tablet f96b08012621442e8c4951d705e4a589 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.238479 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96b08012621442e8c4951d705e4a589. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.240185 29713 tablet_service.cc:1467] Processing CreateTablet for tablet f96b08012621442e8c4951d705e4a589 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.240324 29450 tablet_service.cc:1467] Processing CreateTablet for tablet f96b08012621442e8c4951d705e4a589 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.240445 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96b08012621442e8c4951d705e4a589. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.240531 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96b08012621442e8c4951d705e4a589. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.242640 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 444e705b42e7423b9ee1d187ea1de308 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a1de78f491b94094b3f6674f060f1e22]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:52.242784 8219 tablet_bootstrap.cc:492] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:52.242962 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444e705b42e7423b9ee1d187ea1de308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.243933 8219 tablet_bootstrap.cc:654] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.246071 8219 tablet_bootstrap.cc:492] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:52.246176 8219 ts_tablet_manager.cc:1367] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:52.246618 8219 raft_consensus.cc:348] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.246721 8219 raft_consensus.cc:374] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.246748 8219 raft_consensus.cc:729] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.246870 8219 consensus_queue.cc:260] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.247174 8219 ts_tablet_manager.cc:1398] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:52.248278 7928 consensus_queue.cc:1035] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.249312 8217 tablet_bootstrap.cc:492] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:52.250515 8217 tablet_bootstrap.cc:654] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.251441 8216 tablet_bootstrap.cc:492] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:52.251677 8220 tablet_bootstrap.cc:492] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:52.254078 8216 tablet_bootstrap.cc:654] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.254567 8220 tablet_bootstrap.cc:654] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.256208 8220 tablet_bootstrap.cc:492] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:52.256311 8220 ts_tablet_manager.cc:1367] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:52.256781 8220 raft_consensus.cc:348] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.256883 8220 raft_consensus.cc:374] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.256883 8215 tablet_bootstrap.cc:492] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:52.256916 8220 raft_consensus.cc:729] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.257031 8220 consensus_queue.cc:260] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.257367 8220 ts_tablet_manager.cc:1398] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:52.257400 7813 consensus_queue.cc:1035] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.260038 8218 tablet_bootstrap.cc:492] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:52.265524 8217 tablet_bootstrap.cc:492] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:52.265627 8217 ts_tablet_manager.cc:1367] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240502 14:00:52.265749 8218 tablet_bootstrap.cc:654] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.266069 8217 raft_consensus.cc:348] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.266181 8217 raft_consensus.cc:374] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.266212 8217 raft_consensus.cc:729] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.266357 8217 consensus_queue.cc:260] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.266649 8217 ts_tablet_manager.cc:1398] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:52.266902 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.266711 8215 tablet_bootstrap.cc:654] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.267102 29713 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:00:52.268141 8215 tablet_bootstrap.cc:492] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:52.268280 8215 ts_tablet_manager.cc:1367] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:52.269040 8216 tablet_bootstrap.cc:492] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:52.269138 8216 ts_tablet_manager.cc:1367] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240502 14:00:52.269601 8216 raft_consensus.cc:348] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.269706 8216 raft_consensus.cc:374] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.269738 8216 raft_consensus.cc:729] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.269872 8216 consensus_queue.cc:260] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.270220 8216 ts_tablet_manager.cc:1398] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:52.271030 8215 raft_consensus.cc:348] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.271135 8215 raft_consensus.cc:374] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.271178 8215 raft_consensus.cc:729] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.271313 8215 consensus_queue.cc:260] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.271546 8215 ts_tablet_manager.cc:1398] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 14:00:52.271741 8215 tablet_bootstrap.cc:492] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:52.284396 8215 tablet_bootstrap.cc:654] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.288197 8218 tablet_bootstrap.cc:492] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:52.288690 8218 ts_tablet_manager.cc:1367] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:00:52.289608 8218 raft_consensus.cc:348] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.289713 8218 raft_consensus.cc:374] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.289744 8218 raft_consensus.cc:729] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.289906 8218 consensus_queue.cc:260] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.290345 8218 ts_tablet_manager.cc:1398] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:52.290637 8218 tablet_bootstrap.cc:492] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:52.290932 8215 tablet_bootstrap.cc:492] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:52.291088 8215 ts_tablet_manager.cc:1367] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:00:52.291604 8215 raft_consensus.cc:348] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.291761 8215 raft_consensus.cc:374] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.291836 8215 raft_consensus.cc:729] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.292013 8218 tablet_bootstrap.cc:654] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.292032 8215 consensus_queue.cc:260] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.292411 8215 ts_tablet_manager.cc:1398] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:52.293229 8215 tablet_bootstrap.cc:492] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:52.295761 8215 tablet_bootstrap.cc:654] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.296670 8218 tablet_bootstrap.cc:492] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:52.296754 8218 ts_tablet_manager.cc:1367] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:52.297313 8218 raft_consensus.cc:348] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.297453 8218 raft_consensus.cc:374] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.297508 8218 raft_consensus.cc:729] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.297647 8218 consensus_queue.cc:260] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.297935 8218 ts_tablet_manager.cc:1398] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:52.299355 8215 tablet_bootstrap.cc:492] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:52.299454 8215 ts_tablet_manager.cc:1367] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:52.299880 8215 raft_consensus.cc:348] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.299974 8215 raft_consensus.cc:374] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.300032 8215 raft_consensus.cc:729] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.300150 8215 consensus_queue.cc:260] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.300482 8215 ts_tablet_manager.cc:1398] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:52.335248 29328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:52.335390 29328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:52.337466 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet be35d71a8b9f41a59aa8d0d1b5d09c83
I20240502 14:00:52.338090 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:52 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:52.338274 8098 tablet_replica.cc:330] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:52.338588 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:52 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:52.339016 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet be35d71a8b9f41a59aa8d0d1b5d09c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:52 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:52.339390 8098 raft_consensus.cc:2229] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:52.339512 8098 raft_consensus.cc:2258] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:52.340986 8098 ts_tablet_manager.cc:1875] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:52.343266 8098 ts_tablet_manager.cc:1888] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:52.343324 8098 log.cc:1196] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/be35d71a8b9f41a59aa8d0d1b5d09c83
I20240502 14:00:52.343591 8098 ts_tablet_manager.cc:1909] T be35d71a8b9f41a59aa8d0d1b5d09c83 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:52.344189 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (table rebalancer_dynamic_table_new_name [id=e06e48e5d9ac4f49a28af0ebbc0c3083]) successfully deleted
I20240502 14:00:52.344561 8246 tablet_replica.cc:330] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:52.344777 8246 raft_consensus.cc:2229] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:52.344964 8246 raft_consensus.cc:2258] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:52.345638 8246 ts_tablet_manager.cc:1875] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:52.347568 8246 ts_tablet_manager.cc:1888] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:52.347635 8246 log.cc:1196] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/be35d71a8b9f41a59aa8d0d1b5d09c83
I20240502 14:00:52.347918 8246 ts_tablet_manager.cc:1909] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:52.349323 8245 tablet_replica.cc:330] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:52.349479 8245 raft_consensus.cc:2229] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:52.349511 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (table rebalancer_dynamic_table_new_name [id=e06e48e5d9ac4f49a28af0ebbc0c3083]) successfully deleted
I20240502 14:00:52.349691 8245 raft_consensus.cc:2258] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:52.350307 8245 ts_tablet_manager.cc:1875] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:52.352399 8245 ts_tablet_manager.cc:1888] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:52.352470 8245 log.cc:1196] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/be35d71a8b9f41a59aa8d0d1b5d09c83
I20240502 14:00:52.356137 8245 ts_tablet_manager.cc:1909] T be35d71a8b9f41a59aa8d0d1b5d09c83 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:52.357038 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet be35d71a8b9f41a59aa8d0d1b5d09c83 (table rebalancer_dynamic_table_new_name [id=e06e48e5d9ac4f49a28af0ebbc0c3083]) successfully deleted
I20240502 14:00:52.405766 8061 ts_tablet_manager.cc:902] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:52.406292 8061 tablet_copy_client.cc:308] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:52.406870 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 6b6ced4716f64a3aaad4cd5299e46d13 from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:34141)
I20240502 14:00:52.406983 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 6b6ced4716f64a3aaad4cd5299e46d13 from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:34141: session id = 8973f2527acb48f5ad36589652de73e1-6b6ced4716f64a3aaad4cd5299e46d13
I20240502 14:00:52.407999 29884 tablet_copy_source_session.cc:205] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:52.408838 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6ced4716f64a3aaad4cd5299e46d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.411577 8061 tablet_copy_client.cc:783] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:52.411728 8061 tablet_copy_client.cc:647] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:52.415072 8061 tablet_copy_client.cc:523] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:52.416358 8061 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:52.431577 8282 ts_tablet_manager.cc:902] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:52.432632 8282 tablet_copy_client.cc:308] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:52.433313 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet 86808085d86746a9bea7fe0ea0647d5f from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57511)
I20240502 14:00:52.433420 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet 86808085d86746a9bea7fe0ea0647d5f from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57511: session id = 8973f2527acb48f5ad36589652de73e1-86808085d86746a9bea7fe0ea0647d5f
I20240502 14:00:52.434172 7928 raft_consensus.cc:482] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:52.434275 7928 raft_consensus.cc:504] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.434491 29491 tablet_copy_source_session.cc:205] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:52.434679 7928 leader_election.cc:288] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:52.435261 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444e705b42e7423b9ee1d187ea1de308" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:52.435366 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86808085d86746a9bea7fe0ea0647d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.435405 31859 raft_consensus.cc:2454] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:52.435760 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444e705b42e7423b9ee1d187ea1de308" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:52.435835 29666 leader_election.cc:302] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:52.435880 29470 raft_consensus.cc:2454] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:52.435997 8233 raft_consensus.cc:2781] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:52.436039 8233 raft_consensus.cc:482] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:52.436079 8233 raft_consensus.cc:3037] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.437197 8233 raft_consensus.cc:504] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.437546 8233 leader_election.cc:288] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:52.438252 8282 tablet_copy_client.cc:783] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:52.438540 8282 tablet_copy_client.cc:647] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:52.439302 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:52.439566 8282 tablet_copy_client.cc:523] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:52.441138 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444e705b42e7423b9ee1d187ea1de308" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:52.441257 31860 raft_consensus.cc:3037] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.441895 8282 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:52.442237 31860 raft_consensus.cc:2454] T 444e705b42e7423b9ee1d187ea1de308 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:52.443007 29666 leader_election.cc:302] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:52.443011 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444e705b42e7423b9ee1d187ea1de308" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:52.443145 29470 raft_consensus.cc:3037] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.443456 8233 raft_consensus.cc:2781] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:52.443534 8233 raft_consensus.cc:686] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:52.443651 8233 consensus_queue.cc:237] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.444175 29470 raft_consensus.cc:2454] T 444e705b42e7423b9ee1d187ea1de308 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:52.445839 8061 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:52.446313 8061 tablet_bootstrap.cc:492] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:52.446538 8061 ts_tablet_manager.cc:1367] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.000s
I20240502 14:00:52.446409 29328 catalog_manager.cc:5484] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:52.447185 8061 raft_consensus.cc:348] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:52.447381 8061 raft_consensus.cc:729] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.447575 8061 consensus_queue.cc:260] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:52.447803 31970 tablet_service.cc:1467] Processing CreateTablet for tablet c0b62c5589a245c4a0b7e9edd987ee3a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0]), partition=
I20240502 14:00:52.448067 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b62c5589a245c4a0b7e9edd987ee3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.448078 32101 tablet_service.cc:1467] Processing CreateTablet for tablet c0b62c5589a245c4a0b7e9edd987ee3a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0]), partition=
I20240502 14:00:52.448304 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b62c5589a245c4a0b7e9edd987ee3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.448870 29582 tablet_service.cc:1467] Processing CreateTablet for tablet c0b62c5589a245c4a0b7e9edd987ee3a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0]), partition=
I20240502 14:00:52.449103 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b62c5589a245c4a0b7e9edd987ee3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:52.450054 8216 tablet_bootstrap.cc:492] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:52.450464 8217 tablet_bootstrap.cc:492] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:52.451031 8216 tablet_bootstrap.cc:654] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.451362 8217 tablet_bootstrap.cc:654] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.451651 8061 ts_tablet_manager.cc:1398] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:52.452214 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-6b6ced4716f64a3aaad4cd5299e46d13 received from {username='slave'} at 127.22.55.136:34141
I20240502 14:00:52.452287 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-6b6ced4716f64a3aaad4cd5299e46d13 on tablet 6b6ced4716f64a3aaad4cd5299e46d13 with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:52.452360 8216 tablet_bootstrap.cc:492] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:52.452432 8216 ts_tablet_manager.cc:1367] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:52.453044 8216 raft_consensus.cc:348] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.453169 8216 raft_consensus.cc:374] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.453205 8216 raft_consensus.cc:729] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.453322 8216 consensus_queue.cc:260] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.453608 8216 ts_tablet_manager.cc:1398] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:52.455832 8217 tablet_bootstrap.cc:492] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:52.455920 8217 ts_tablet_manager.cc:1367] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:52.456434 8217 raft_consensus.cc:348] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.456552 8217 raft_consensus.cc:374] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.456586 8217 raft_consensus.cc:729] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.456692 8217 consensus_queue.cc:260] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.456941 8217 ts_tablet_manager.cc:1398] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:52.458037 8288 tablet_bootstrap.cc:492] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:52.459002 8288 tablet_bootstrap.cc:654] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:52.463425 8288 tablet_bootstrap.cc:492] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:52.463510 8288 ts_tablet_manager.cc:1367] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:52.464017 8288 raft_consensus.cc:348] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.464272 8288 raft_consensus.cc:374] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:52.465098 8288 raft_consensus.cc:729] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.465291 8288 consensus_queue.cc:260] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.465621 8288 ts_tablet_manager.cc:1398] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:52.465837 7928 raft_consensus.cc:482] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:52.465921 7928 raft_consensus.cc:504] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.466272 7928 leader_election.cc:288] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:52.466568 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96b08012621442e8c4951d705e4a589" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:52.466703 29470 raft_consensus.cc:2454] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:52.466912 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96b08012621442e8c4951d705e4a589" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:52.467037 29864 raft_consensus.cc:2454] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:52.467099 29668 leader_election.cc:302] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:52.467291 7928 raft_consensus.cc:2781] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:52.467360 7928 raft_consensus.cc:482] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:52.467408 7928 raft_consensus.cc:3037] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.468477 7928 raft_consensus.cc:504] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.468840 7928 leader_election.cc:288] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:52.468991 7621 raft_consensus.cc:482] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:52.469069 7621 raft_consensus.cc:504] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.469167 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96b08012621442e8c4951d705e4a589" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:52.469167 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96b08012621442e8c4951d705e4a589" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:52.469271 29471 raft_consensus.cc:3037] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.469271 29864 raft_consensus.cc:3037] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.469406 7621 leader_election.cc:288] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:52.469914 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65229e977b8543639fb27bbbc3b5e797" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:52.470033 31990 raft_consensus.cc:2454] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:52.470299 29864 raft_consensus.cc:2454] T f96b08012621442e8c4951d705e4a589 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:52.470299 29471 raft_consensus.cc:2454] T f96b08012621442e8c4951d705e4a589 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:52.470371 32056 leader_election.cc:302] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:52.470687 7621 raft_consensus.cc:2781] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:52.470739 7621 raft_consensus.cc:482] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:52.470777 7621 raft_consensus.cc:3037] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.470904 7431 raft_consensus.cc:482] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:52.470978 7431 raft_consensus.cc:504] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.471091 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65229e977b8543639fb27bbbc3b5e797" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:52.471208 29729 raft_consensus.cc:2454] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:52.471347 7431 leader_election.cc:288] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:52.471632 29667 leader_election.cc:302] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:52.471812 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b62c5589a245c4a0b7e9edd987ee3a" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:52.471918 31990 raft_consensus.cc:2454] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:52.472172 29535 leader_election.cc:302] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:52.472393 7431 raft_consensus.cc:2781] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:52.472455 7431 raft_consensus.cc:482] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:52.472493 7431 raft_consensus.cc:3037] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.473460 7431 raft_consensus.cc:504] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.473778 7431 leader_election.cc:288] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:52.474897 7621 raft_consensus.cc:504] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.475129 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b62c5589a245c4a0b7e9edd987ee3a" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:52.475211 7621 leader_election.cc:288] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:52.475232 31990 raft_consensus.cc:3037] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.475587 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65229e977b8543639fb27bbbc3b5e797" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:52.475685 29729 raft_consensus.cc:3037] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.476225 31990 raft_consensus.cc:2454] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:52.476508 29535 leader_election.cc:302] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:52.476675 29729 raft_consensus.cc:2454] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:52.476819 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b62c5589a245c4a0b7e9edd987ee3a" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:52.476866 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65229e977b8543639fb27bbbc3b5e797" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:52.476923 32119 raft_consensus.cc:3037] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.476954 31992 raft_consensus.cc:3037] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:52.477675 32119 raft_consensus.cc:2454] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:52.477823 31992 raft_consensus.cc:2454] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:52.477901 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b62c5589a245c4a0b7e9edd987ee3a" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:52.477998 32122 raft_consensus.cc:2362] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1. Re-sending same reply.
I20240502 14:00:52.478039 32057 leader_election.cc:302] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:52.478341 7928 raft_consensus.cc:2781] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:52.478433 7928 raft_consensus.cc:686] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:52.478559 7431 raft_consensus.cc:2781] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:52.478557 7928 consensus_queue.cc:237] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:52.478641 7431 raft_consensus.cc:686] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:52.478770 7431 consensus_queue.cc:237] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:52.479972 7621 raft_consensus.cc:2781] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:52.480059 7621 raft_consensus.cc:686] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:52.480187 7621 consensus_queue.cc:237] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:52.483283 29320 catalog_manager.cc:5484] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:52.485095 29328 catalog_manager.cc:5484] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:52.485411 8282 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:52.485724 8282 tablet_bootstrap.cc:492] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:52.485904 8282 ts_tablet_manager.cc:1367] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.005s
I20240502 14:00:52.486374 8282 raft_consensus.cc:348] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:52.486371 29322 catalog_manager.cc:5484] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:52.486485 8282 raft_consensus.cc:729] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:52.486632 8282 consensus_queue.cc:260] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:52.487015 8282 ts_tablet_manager.cc:1398] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:00:52.487602 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-86808085d86746a9bea7fe0ea0647d5f received from {username='slave'} at 127.22.55.136:57511
I20240502 14:00:52.487680 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-86808085d86746a9bea7fe0ea0647d5f on tablet 86808085d86746a9bea7fe0ea0647d5f with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:52.513123 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:52.525796 29450 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:52.528365 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.529664 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:52.531157 29582 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.538229 32101 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:52.546461 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.561772 29842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:52.594286 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:52.598745 32122 raft_consensus.cc:1261] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:52.598809 31990 raft_consensus.cc:1261] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:52.599279 7431 consensus_queue.cc:1035] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.599617 7431 consensus_queue.cc:1035] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.603039 29320 catalog_manager.cc:5576] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137) does not have the latest schema for tablet c0b62c5589a245c4a0b7e9edd987ee3a (table rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0]). Expected version 1 got 0
I20240502 14:00:52.603950 8124 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:52.604342 8124 tablet.cc:1620] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:52.606086 29320 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet c0b62c5589a245c4a0b7e9edd987ee3a (table rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0]). Expected version 1 got 0
I20240502 14:00:52.609282 8321 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:52.610793 8321 tablet.cc:1620] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:52.612890 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0] alter complete (version 1)
I20240502 14:00:52.614125 8322 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:52.615257 29322 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet c0b62c5589a245c4a0b7e9edd987ee3a (table rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0]). Expected version 1 got 0
I20240502 14:00:52.615790 8322 tablet.cc:1620] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:52.717428 29328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:52.738169 8124 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:52.738574 8124 tablet.cc:1620] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:52.738940 8322 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:52.739224 8322 tablet.cc:1620] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:52.745718 8321 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:52.746009 8321 tablet.cc:1620] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:52.747653 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=a641b3ff9c9a4037b62c4297ee898ab0] alter complete (version 2)
I20240502 14:00:52.775974 29842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:52.781214 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:52.781608 29450 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:52.786213 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:52.789132 32101 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:52.791818 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.815016 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.821333 29582 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:52.833786 32254 raft_consensus.cc:1203] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:52.837661 8233 consensus_queue.cc:1035] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.841809 7928 consensus_queue.cc:1035] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.853947 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:52.859762 8124 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:52.860311 8322 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:52.861131 8321 tablet.cc:1722] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:52.866173 29324 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=a641b3ff9c9a4037b62c4297ee898ab0] alter complete (version 3)
I20240502 14:00:52.885095 29729 consensus_queue.cc:237] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:52.886224 8371 consensus_queue.cc:1035] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.886353 29471 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:52.886734 32255 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:52.886879 8371 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:52.887198 8233 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:52.891321 7928 consensus_queue.cc:1035] T f96b08012621442e8c4951d705e4a589 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.892000 8233 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.894755 32255 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.895402 29471 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.895460 29320 catalog_manager.cc:5484] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:52.900337 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cc4394003840a5b1d59b0fe72c5164 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:52.900837 8403 tablet_replica.cc:330] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:52.901072 8403 raft_consensus.cc:2229] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:52.901288 8403 raft_consensus.cc:2258] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:52.902007 8403 ts_tablet_manager.cc:1875] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:52.904175 8403 ts_tablet_manager.cc:1888] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:52.904314 8403 log.cc:1196] T 92cc4394003840a5b1d59b0fe72c5164 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/92cc4394003840a5b1d59b0fe72c5164
I20240502 14:00:52.904930 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 92cc4394003840a5b1d59b0fe72c5164 (table rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]) successfully deleted
I20240502 14:00:52.919739 32255 raft_consensus.cc:1203] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:52.926226 7237 consensus_queue.cc:1035] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.929780 7621 consensus_queue.cc:1035] T 65229e977b8543639fb27bbbc3b5e797 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:52.970234 29328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:52.970382 29328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:52.972853 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet c0b62c5589a245c4a0b7e9edd987ee3a
I20240502 14:00:52.973320 31970 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b62c5589a245c4a0b7e9edd987ee3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:52 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:52.974607 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b62c5589a245c4a0b7e9edd987ee3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:52 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:52.976891 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b62c5589a245c4a0b7e9edd987ee3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:52 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:52.977144 8403 tablet_replica.cc:330] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:52.977269 8450 tablet_replica.cc:330] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:52.977298 8403 raft_consensus.cc:2229] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:52.977432 8450 raft_consensus.cc:2229] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:52.977531 8450 raft_consensus.cc:2258] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:52.977537 8403 raft_consensus.cc:2258] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:52.978107 8450 ts_tablet_manager.cc:1875] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:52.978811 8403 ts_tablet_manager.cc:1875] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:52.981228 8403 ts_tablet_manager.cc:1888] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:52.981632 8403 log.cc:1196] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0b62c5589a245c4a0b7e9edd987ee3a
I20240502 14:00:52.981626 8450 ts_tablet_manager.cc:1888] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:52.981705 8450 log.cc:1196] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/c0b62c5589a245c4a0b7e9edd987ee3a
I20240502 14:00:52.982018 8450 ts_tablet_manager.cc:1909] T c0b62c5589a245c4a0b7e9edd987ee3a P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:52.982298 8403 ts_tablet_manager.cc:1909] T c0b62c5589a245c4a0b7e9edd987ee3a P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:52.983352 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet c0b62c5589a245c4a0b7e9edd987ee3a (table rebalancer_dynamic_table_new_name [id=a641b3ff9c9a4037b62c4297ee898ab0]) successfully deleted
I20240502 14:00:52.986737 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet c0b62c5589a245c4a0b7e9edd987ee3a (table rebalancer_dynamic_table_new_name [id=a641b3ff9c9a4037b62c4297ee898ab0]) successfully deleted
I20240502 14:00:52.993350 8449 tablet_replica.cc:330] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:52.993760 8449 raft_consensus.cc:2229] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:52.993968 8449 raft_consensus.cc:2258] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:52.997354 8449 ts_tablet_manager.cc:1875] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:53.000982 8449 ts_tablet_manager.cc:1888] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:53.001159 8449 log.cc:1196] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/c0b62c5589a245c4a0b7e9edd987ee3a
I20240502 14:00:53.005434 8449 ts_tablet_manager.cc:1909] T c0b62c5589a245c4a0b7e9edd987ee3a P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:53.006323 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet c0b62c5589a245c4a0b7e9edd987ee3a (table rebalancer_dynamic_table_new_name [id=a641b3ff9c9a4037b62c4297ee898ab0]) successfully deleted
I20240502 14:00:53.045434 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:53.052279 29450 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:00:53.054461 31970 tablet_service.cc:1467] Processing CreateTablet for tablet f5f316f08f5041a4907ad768c28ae30f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.054761 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f316f08f5041a4907ad768c28ae30f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.058593 31840 tablet_service.cc:1467] Processing CreateTablet for tablet a63de5b3f35e4486859c9361925689d9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.058729 29842 tablet_service.cc:1467] Processing CreateTablet for tablet f5f316f08f5041a4907ad768c28ae30f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.058785 29843 tablet_service.cc:1467] Processing CreateTablet for tablet aeab77cba6cf431e89141d4154c41a9b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.058898 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63de5b3f35e4486859c9361925689d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.058997 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f316f08f5041a4907ad768c28ae30f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.059365 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeab77cba6cf431e89141d4154c41a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.061412 29582 tablet_service.cc:1467] Processing CreateTablet for tablet f5f316f08f5041a4907ad768c28ae30f (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.061702 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:53.061852 29580 tablet_service.cc:1467] Processing CreateTablet for tablet a63de5b3f35e4486859c9361925689d9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.062119 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63de5b3f35e4486859c9361925689d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.063908 32101 tablet_service.cc:1467] Processing CreateTablet for tablet a63de5b3f35e4486859c9361925689d9 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.063992 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f316f08f5041a4907ad768c28ae30f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.064059 31970 tablet_service.cc:1467] Processing CreateTablet for tablet aeab77cba6cf431e89141d4154c41a9b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.064173 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63de5b3f35e4486859c9361925689d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.064265 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeab77cba6cf431e89141d4154c41a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.071324 32101 tablet_service.cc:1467] Processing CreateTablet for tablet aeab77cba6cf431e89141d4154c41a9b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=d49ba0808a3044aeb472aaface53a517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.071581 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeab77cba6cf431e89141d4154c41a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.077096 8465 tablet_bootstrap.cc:492] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:53.079339 8463 tablet_bootstrap.cc:492] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:53.079551 29580 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:53.080855 8463 tablet_bootstrap.cc:654] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.081120 8465 tablet_bootstrap.cc:654] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.081974 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:53.086421 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.086701 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:53.088654 32101 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:53.091526 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.106333 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:53.115795 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 1516806f7adf4f928f27050d7947d9da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146]), partition=
I20240502 14:00:53.115881 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 1516806f7adf4f928f27050d7947d9da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146]), partition=
I20240502 14:00:53.115924 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 1516806f7adf4f928f27050d7947d9da (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146]), partition=
I20240502 14:00:53.116102 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1516806f7adf4f928f27050d7947d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.116154 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1516806f7adf4f928f27050d7947d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.116153 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1516806f7adf4f928f27050d7947d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.128057 8463 tablet_bootstrap.cc:492] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:53.131103 8463 ts_tablet_manager.cc:1367] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.000s
I20240502 14:00:53.131692 8463 raft_consensus.cc:348] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.134318 8463 raft_consensus.cc:374] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.134428 8463 raft_consensus.cc:729] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.134618 8463 consensus_queue.cc:260] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.135035 8463 ts_tablet_manager.cc:1398] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:53.137102 8463 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:53.139884 8463 tablet_bootstrap.cc:654] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.140369 8462 tablet_bootstrap.cc:492] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:53.141217 8464 tablet_bootstrap.cc:492] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:53.142058 8464 tablet_bootstrap.cc:654] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.142360 8466 tablet_bootstrap.cc:492] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:53.143311 8466 tablet_bootstrap.cc:654] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.143675 8465 tablet_bootstrap.cc:492] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:53.143819 8465 ts_tablet_manager.cc:1367] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.000s
I20240502 14:00:53.144178 8462 tablet_bootstrap.cc:654] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.144369 8465 raft_consensus.cc:348] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.144557 8465 raft_consensus.cc:374] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.144640 8465 raft_consensus.cc:729] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.144824 8465 consensus_queue.cc:260] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.145292 8469 tablet_bootstrap.cc:492] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:53.145663 8465 ts_tablet_manager.cc:1398] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:53.145885 8465 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:53.146984 8465 tablet_bootstrap.cc:654] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.146984 8469 tablet_bootstrap.cc:654] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.147661 8468 tablet_bootstrap.cc:492] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:53.148546 8468 tablet_bootstrap.cc:654] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.152714 8464 tablet_bootstrap.cc:492] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:53.152848 8464 ts_tablet_manager.cc:1367] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:53.153311 8464 raft_consensus.cc:348] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.153474 8464 raft_consensus.cc:374] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.153738 8464 raft_consensus.cc:729] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.154220 8466 tablet_bootstrap.cc:492] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:53.154247 8464 consensus_queue.cc:260] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.154371 8466 ts_tablet_manager.cc:1367] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:00:53.154837 8466 raft_consensus.cc:348] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.155036 8466 raft_consensus.cc:374] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.155108 8466 raft_consensus.cc:729] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.155488 8464 ts_tablet_manager.cc:1398] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:53.155712 8464 tablet_bootstrap.cc:492] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:53.156862 8464 tablet_bootstrap.cc:654] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.158041 8466 consensus_queue.cc:260] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.158670 8466 ts_tablet_manager.cc:1398] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:53.159118 8465 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:53.159210 8465 ts_tablet_manager.cc:1367] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240502 14:00:53.159708 8465 raft_consensus.cc:348] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.159829 8465 raft_consensus.cc:374] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.159863 8465 raft_consensus.cc:729] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.159970 8465 consensus_queue.cc:260] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.160329 8465 ts_tablet_manager.cc:1398] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:53.161013 8465 tablet_bootstrap.cc:492] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:53.161981 8465 tablet_bootstrap.cc:654] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.162199 8469 tablet_bootstrap.cc:492] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:53.162313 8469 ts_tablet_manager.cc:1367] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:53.162926 8469 raft_consensus.cc:348] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.163106 8469 raft_consensus.cc:374] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.163165 8469 raft_consensus.cc:729] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.163336 8469 consensus_queue.cc:260] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.164518 8462 tablet_bootstrap.cc:492] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:53.164655 8462 ts_tablet_manager.cc:1367] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240502 14:00:53.165201 8462 raft_consensus.cc:348] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.165344 8469 ts_tablet_manager.cc:1398] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:53.165380 8462 raft_consensus.cc:374] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.165433 8462 raft_consensus.cc:729] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.167263 8462 consensus_queue.cc:260] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.168097 8462 ts_tablet_manager.cc:1398] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:53.168911 8464 tablet_bootstrap.cc:492] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:53.169054 8464 ts_tablet_manager.cc:1367] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:00:53.169100 8463 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:53.169150 8468 tablet_bootstrap.cc:492] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:53.169186 8463 ts_tablet_manager.cc:1367] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240502 14:00:53.169279 8468 ts_tablet_manager.cc:1367] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240502 14:00:53.169541 8464 raft_consensus.cc:348] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.169646 8464 raft_consensus.cc:374] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.169600 8463 raft_consensus.cc:348] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.169675 8464 raft_consensus.cc:729] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.169694 8463 raft_consensus.cc:374] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.169728 8463 raft_consensus.cc:729] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.169818 8464 consensus_queue.cc:260] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.169860 8463 consensus_queue.cc:260] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.169812 8468 raft_consensus.cc:348] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.169939 8468 raft_consensus.cc:374] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.169979 8468 raft_consensus.cc:729] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.170122 8463 ts_tablet_manager.cc:1398] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:53.170118 8468 consensus_queue.cc:260] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.170472 8468 ts_tablet_manager.cc:1398] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:53.170951 8464 ts_tablet_manager.cc:1398] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:53.177029 8462 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:53.178138 8462 tablet_bootstrap.cc:654] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.179327 8465 tablet_bootstrap.cc:492] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:53.179801 8465 ts_tablet_manager.cc:1367] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240502 14:00:53.180506 8465 raft_consensus.cc:348] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.180673 8465 raft_consensus.cc:374] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.180752 8465 raft_consensus.cc:729] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.181525 8465 consensus_queue.cc:260] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.182195 8465 ts_tablet_manager.cc:1398] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:00:53.183424 8462 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:53.183495 8462 ts_tablet_manager.cc:1367] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:53.183880 8462 raft_consensus.cc:348] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.183969 8462 raft_consensus.cc:374] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.183996 8462 raft_consensus.cc:729] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.184108 8462 consensus_queue.cc:260] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.184352 8462 ts_tablet_manager.cc:1398] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:53.201573 7237 raft_consensus.cc:482] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:53.201678 7237 raft_consensus.cc:504] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.202054 7237 leader_election.cc:288] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:53.202335 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63de5b3f35e4486859c9361925689d9" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:53.202409 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63de5b3f35e4486859c9361925689d9" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:53.202469 31860 raft_consensus.cc:2454] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:53.202533 29600 raft_consensus.cc:2454] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:53.202730 32056 leader_election.cc:302] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:53.202934 7237 raft_consensus.cc:2781] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:53.202986 7237 raft_consensus.cc:482] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:53.203023 7237 raft_consensus.cc:3037] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.204842 7237 raft_consensus.cc:504] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.205250 7237 leader_election.cc:288] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:53.205578 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63de5b3f35e4486859c9361925689d9" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:53.205690 31860 raft_consensus.cc:3037] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.205729 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63de5b3f35e4486859c9361925689d9" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:53.205839 29600 raft_consensus.cc:3037] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.206482 31860 raft_consensus.cc:2454] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:53.206627 29600 raft_consensus.cc:2454] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:53.206799 32056 leader_election.cc:302] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:53.207070 7621 raft_consensus.cc:2781] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:53.207136 7621 raft_consensus.cc:686] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:53.207229 7621 consensus_queue.cc:237] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.209496 29320 catalog_manager.cc:5484] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:53.213138 29471 consensus_queue.cc:237] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.214286 32254 raft_consensus.cc:1261] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:53.214294 29729 raft_consensus.cc:1261] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:53.214663 8448 consensus_queue.cc:1035] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:53.214773 8152 consensus_queue.cc:1035] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:53.216337 8152 raft_consensus.cc:2932] T 86808085d86746a9bea7fe0ea0647d5f P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.217207 29729 raft_consensus.cc:2932] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.218956 32254 raft_consensus.cc:2932] T 86808085d86746a9bea7fe0ea0647d5f P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.220849 29328 catalog_manager.cc:5484] T 86808085d86746a9bea7fe0ea0647d5f P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 4 to 5, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.222714 7621 raft_consensus.cc:482] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:53.222801 7621 raft_consensus.cc:504] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.223172 7621 leader_election.cc:288] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:53.224848 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab77cba6cf431e89141d4154c41a9b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:53.224866 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab77cba6cf431e89141d4154c41a9b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:53.224970 31990 raft_consensus.cc:2454] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:53.225021 29863 raft_consensus.cc:2454] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:53.225304 32056 leader_election.cc:302] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:53.225466 7621 raft_consensus.cc:2781] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:53.225522 7621 raft_consensus.cc:482] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:53.225574 7621 raft_consensus.cc:3037] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.226632 7621 raft_consensus.cc:504] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.226992 7621 leader_election.cc:288] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:53.227067 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 86808085d86746a9bea7fe0ea0647d5f with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:53.227284 8403 tablet_replica.cc:330] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:53.227372 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab77cba6cf431e89141d4154c41a9b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:53.227437 8403 raft_consensus.cc:2229] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.227478 29863 raft_consensus.cc:3037] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.227583 8403 raft_consensus.cc:2258] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.228248 8403 ts_tablet_manager.cc:1875] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:53.228487 29863 raft_consensus.cc:2454] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:53.228763 32057 leader_election.cc:302] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:53.228925 7621 raft_consensus.cc:2781] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:53.229008 7621 raft_consensus.cc:686] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:53.229130 7621 consensus_queue.cc:237] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:53.230118 8403 ts_tablet_manager.cc:1888] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:53.230167 8403 log.cc:1196] T 86808085d86746a9bea7fe0ea0647d5f P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/86808085d86746a9bea7fe0ea0647d5f
I20240502 14:00:53.231089 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 86808085d86746a9bea7fe0ea0647d5f (table rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]) successfully deleted
I20240502 14:00:53.231298 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab77cba6cf431e89141d4154c41a9b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:53.231420 31990 raft_consensus.cc:3037] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.232012 29322 catalog_manager.cc:5484] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:53.232194 31990 raft_consensus.cc:2454] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:53.277731 8311 raft_consensus.cc:482] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:53.277865 8311 raft_consensus.cc:504] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.278291 8311 leader_election.cc:288] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:53.278808 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f316f08f5041a4907ad768c28ae30f" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:53.278822 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f316f08f5041a4907ad768c28ae30f" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:53.278986 29863 raft_consensus.cc:2454] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:53.278985 31990 raft_consensus.cc:2454] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:53.279713 29536 leader_election.cc:302] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:53.280637 7431 raft_consensus.cc:2781] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:53.280709 7431 raft_consensus.cc:482] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:53.280748 7431 raft_consensus.cc:3037] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.281877 7431 raft_consensus.cc:504] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.282210 7431 leader_election.cc:288] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:53.282722 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f316f08f5041a4907ad768c28ae30f" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:53.282867 31990 raft_consensus.cc:3037] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.283663 31990 raft_consensus.cc:2454] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:53.284865 29535 leader_election.cc:302] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:53.285202 7191 raft_consensus.cc:2781] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:53.285274 7191 raft_consensus.cc:686] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:53.285418 7191 consensus_queue.cc:237] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.286904 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f316f08f5041a4907ad768c28ae30f" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:53.287029 29863 raft_consensus.cc:3037] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.287794 29863 raft_consensus.cc:2454] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:53.289381 29320 catalog_manager.cc:5484] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:53.339838 8152 raft_consensus.cc:482] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:53.339958 8152 raft_consensus.cc:504] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.340359 8152 leader_election.cc:288] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:53.340848 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1516806f7adf4f928f27050d7947d9da" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:53.340975 32254 raft_consensus.cc:2454] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:53.341325 29406 leader_election.cc:302] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:53.341626 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1516806f7adf4f928f27050d7947d9da" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:53.341751 32119 raft_consensus.cc:2454] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:53.342214 8152 raft_consensus.cc:2781] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:53.342272 8152 raft_consensus.cc:482] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:53.342309 8152 raft_consensus.cc:3037] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.343359 8152 raft_consensus.cc:504] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.343698 8152 leader_election.cc:288] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:53.344782 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1516806f7adf4f928f27050d7947d9da" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:53.344883 32254 raft_consensus.cc:3037] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.345942 32254 raft_consensus.cc:2454] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:53.346377 29406 leader_election.cc:302] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:53.347730 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1516806f7adf4f928f27050d7947d9da" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:53.347854 32119 raft_consensus.cc:3037] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.348752 32119 raft_consensus.cc:2454] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:53.350543 8152 raft_consensus.cc:2781] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:53.350626 8152 raft_consensus.cc:686] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:53.350750 8152 consensus_queue.cc:237] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.355258 29329 catalog_manager.cc:5484] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:53.374835 29450 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:53.380082 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:53.389294 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:53.391862 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.408672 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:53.418677 32101 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:00:53.423111 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.439797 29580 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.468618 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:53.473032 32255 raft_consensus.cc:1261] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:53.473615 32119 raft_consensus.cc:1261] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:53.473889 8152 consensus_queue.cc:1035] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.475561 8152 consensus_queue.cc:1035] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.479478 29329 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 1516806f7adf4f928f27050d7947d9da (table rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146]). Expected version 1 got 0
I20240502 14:00:53.481524 29326 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 1516806f7adf4f928f27050d7947d9da (table rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146]). Expected version 1 got 0
I20240502 14:00:53.483382 8574 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:53.483671 8573 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:53.484010 8573 tablet.cc:1620] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:53.484032 29326 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet 1516806f7adf4f928f27050d7947d9da (table rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146]). Expected version 1 got 0
I20240502 14:00:53.484462 8574 tablet.cc:1620] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:53.484941 29329 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 1516806f7adf4f928f27050d7947d9da (table rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146]). Expected version 1 got 0
I20240502 14:00:53.484974 8572 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:53.485283 8572 tablet.cc:1620] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:53.497471 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146] alter complete (version 1)
I20240502 14:00:53.497426 29863 consensus_queue.cc:237] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.498684 29471 raft_consensus.cc:1261] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:53.498927 32254 raft_consensus.cc:1261] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:53.499388 8067 consensus_queue.cc:1035] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:53.500763 7551 consensus_queue.cc:1035] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:53.503352 8067 raft_consensus.cc:2932] T 6b6ced4716f64a3aaad4cd5299e46d13 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.503783 32254 raft_consensus.cc:2932] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.508689 29321 catalog_manager.cc:5484] T 6b6ced4716f64a3aaad4cd5299e46d13 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.512873 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6ced4716f64a3aaad4cd5299e46d13 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:53.512745 29471 raft_consensus.cc:2932] T 6b6ced4716f64a3aaad4cd5299e46d13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:53.515055 8587 tablet_replica.cc:330] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:53.515362 8587 raft_consensus.cc:2229] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.515588 8587 raft_consensus.cc:2258] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.516139 8587 ts_tablet_manager.cc:1875] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:53.518589 8587 ts_tablet_manager.cc:1888] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:00:53.518707 8587 log.cc:1196] T 6b6ced4716f64a3aaad4cd5299e46d13 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b6ced4716f64a3aaad4cd5299e46d13
I20240502 14:00:53.519841 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 6b6ced4716f64a3aaad4cd5299e46d13 (table rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]) successfully deleted
I20240502 14:00:53.603590 29322 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:53.608909 8572 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:53.609269 8572 tablet.cc:1620] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:53.609441 8574 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:53.609717 8574 tablet.cc:1620] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:53.610221 8573 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:53.610497 8573 tablet.cc:1620] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:53.617733 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=cbbb530872cf4ea7be7e050059219146] alter complete (version 2)
I20240502 14:00:53.663460 7237 consensus_queue.cc:1035] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.667367 7621 consensus_queue.cc:1035] T aeab77cba6cf431e89141d4154c41a9b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.677385 7621 consensus_queue.cc:1035] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.680595 7621 consensus_queue.cc:1035] T a63de5b3f35e4486859c9361925689d9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.699728 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:53.702731 29450 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:53.703359 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.720556 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:53.720736 32101 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:00:53.730933 8572 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:53.733230 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:53.734474 8573 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:53.737862 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.739440 8574 tablet.cc:1722] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:53.744315 29326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=cbbb530872cf4ea7be7e050059219146] alter complete (version 3)
I20240502 14:00:53.753934 29580 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:53.758116 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:53.836493 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:53.843322 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 7ede8318b945487cae52b2140358b2d9 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.843380 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 7ede8318b945487cae52b2140358b2d9 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.843684 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ede8318b945487cae52b2140358b2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.843693 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ede8318b945487cae52b2140358b2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.846510 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 7ede8318b945487cae52b2140358b2d9 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.846524 29450 tablet_service.cc:1467] Processing CreateTablet for tablet bfad98900ade46899e78ae1ba7a54dbc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.846554 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3fb8ad72148328d8f9d5750c0bd14 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.846634 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3fb8ad72148328d8f9d5750c0bd14 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.846781 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ede8318b945487cae52b2140358b2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.846809 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfad98900ade46899e78ae1ba7a54dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.847246 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3fb8ad72148328d8f9d5750c0bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.847260 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3fb8ad72148328d8f9d5750c0bd14 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.847466 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3fb8ad72148328d8f9d5750c0bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.850335 8658 tablet_bootstrap.cc:492] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:53.850745 8659 tablet_bootstrap.cc:492] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:53.851671 8658 tablet_bootstrap.cc:654] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.851822 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3fb8ad72148328d8f9d5750c0bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.851985 8659 tablet_bootstrap.cc:654] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.852663 8658 tablet_bootstrap.cc:492] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:53.852751 8658 ts_tablet_manager.cc:1367] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:53.853124 8659 tablet_bootstrap.cc:492] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:53.853137 8658 raft_consensus.cc:348] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.853246 8658 raft_consensus.cc:374] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.853293 8659 ts_tablet_manager.cc:1367] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:53.853302 8658 raft_consensus.cc:729] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.853440 8658 consensus_queue.cc:260] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.853763 8658 ts_tablet_manager.cc:1398] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:53.853814 8659 raft_consensus.cc:348] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.853955 8659 raft_consensus.cc:374] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.854010 8659 raft_consensus.cc:729] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.854208 8659 consensus_queue.cc:260] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.854588 8659 ts_tablet_manager.cc:1398] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:53.855093 29323 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:53.855232 29323 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:53.856185 29713 tablet_service.cc:1467] Processing CreateTablet for tablet bfad98900ade46899e78ae1ba7a54dbc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.856448 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfad98900ade46899e78ae1ba7a54dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.856776 8660 tablet_bootstrap.cc:492] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:53.857443 8659 tablet_bootstrap.cc:492] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:53.858525 8659 tablet_bootstrap.cc:654] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.858636 8657 tablet_bootstrap.cc:492] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:53.858963 29323 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 1516806f7adf4f928f27050d7947d9da
I20240502 14:00:53.858974 29843 tablet_service.cc:1467] Processing CreateTablet for tablet bfad98900ade46899e78ae1ba7a54dbc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=863f596b0ca14968bd2d48375f60013d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:53.859246 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfad98900ade46899e78ae1ba7a54dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.860036 7431 consensus_queue.cc:1035] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.860080 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 1516806f7adf4f928f27050d7947d9da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:53 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:53.862046 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 1516806f7adf4f928f27050d7947d9da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:53 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:53.862356 8659 tablet_bootstrap.cc:492] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:53.862440 8659 ts_tablet_manager.cc:1367] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:53.862877 8659 raft_consensus.cc:348] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.862973 8659 raft_consensus.cc:374] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.863010 8659 raft_consensus.cc:729] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.863135 8659 consensus_queue.cc:260] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.863408 8659 ts_tablet_manager.cc:1398] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:53.864377 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet 1516806f7adf4f928f27050d7947d9da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:53 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:53.865456 8664 tablet_replica.cc:330] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:53.865649 8664 raft_consensus.cc:2229] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.865839 8664 raft_consensus.cc:2258] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.866590 8664 ts_tablet_manager.cc:1875] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:53.867666 8660 tablet_bootstrap.cc:654] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.870313 8657 tablet_bootstrap.cc:654] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.871003 8664 ts_tablet_manager.cc:1888] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:53.871285 8664 log.cc:1196] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/1516806f7adf4f928f27050d7947d9da
I20240502 14:00:53.867079 8658 tablet_bootstrap.cc:492] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:53.871637 8664 ts_tablet_manager.cc:1909] T 1516806f7adf4f928f27050d7947d9da P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:53.871994 8657 tablet_bootstrap.cc:492] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:53.872339 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 1516806f7adf4f928f27050d7947d9da (table rebalancer_dynamic_table_new_name [id=cbbb530872cf4ea7be7e050059219146]) successfully deleted
I20240502 14:00:53.872567 8658 tablet_bootstrap.cc:654] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.872831 8660 tablet_bootstrap.cc:492] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:53.872924 8660 ts_tablet_manager.cc:1367] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:00:53.873034 8657 ts_tablet_manager.cc:1367] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240502 14:00:53.873203 8663 tablet_bootstrap.cc:492] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:53.873533 8657 raft_consensus.cc:348] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.873649 8657 raft_consensus.cc:374] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.874120 8663 tablet_bootstrap.cc:654] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.874091 8660 raft_consensus.cc:348] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.874258 8657 raft_consensus.cc:729] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.874410 8657 consensus_queue.cc:260] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:53.874689 8657 ts_tablet_manager.cc:1398] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:53.874194 8660 raft_consensus.cc:374] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.874891 8660 raft_consensus.cc:729] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.875082 8660 consensus_queue.cc:260] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.877354 7431 consensus_queue.cc:1035] T f5f316f08f5041a4907ad768c28ae30f P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:53.879469 8663 tablet_bootstrap.cc:492] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:53.879619 8663 ts_tablet_manager.cc:1367] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:53.880019 8663 raft_consensus.cc:348] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.880439 8663 raft_consensus.cc:374] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.880457 8660 ts_tablet_manager.cc:1398] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:53.880555 8663 raft_consensus.cc:729] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.880748 8663 consensus_queue.cc:260] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.880807 8658 tablet_bootstrap.cc:492] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:53.880894 8658 ts_tablet_manager.cc:1367] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240502 14:00:53.876152 8667 tablet_replica.cc:330] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:53.881135 8663 ts_tablet_manager.cc:1398] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:53.881455 8658 raft_consensus.cc:348] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.881590 8658 raft_consensus.cc:374] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.881641 8658 raft_consensus.cc:729] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.881791 8667 raft_consensus.cc:2229] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.881945 8667 raft_consensus.cc:2258] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.881928 8658 consensus_queue.cc:260] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.882807 8667 ts_tablet_manager.cc:1875] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:53.883101 8658 ts_tablet_manager.cc:1398] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:53.884232 8672 tablet_replica.cc:330] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:53.888742 8660 tablet_bootstrap.cc:492] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:53.884459 8672 raft_consensus.cc:2229] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:53.889043 8672 raft_consensus.cc:2258] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.889633 8672 ts_tablet_manager.cc:1875] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:53.892509 8660 tablet_bootstrap.cc:654] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.893162 8667 ts_tablet_manager.cc:1888] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:53.893556 8672 ts_tablet_manager.cc:1888] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:53.893633 8672 log.cc:1196] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/1516806f7adf4f928f27050d7947d9da
I20240502 14:00:53.894003 8672 ts_tablet_manager.cc:1909] T 1516806f7adf4f928f27050d7947d9da P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:53.894536 8667 log.cc:1196] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/1516806f7adf4f928f27050d7947d9da
I20240502 14:00:53.894726 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 1516806f7adf4f928f27050d7947d9da (table rebalancer_dynamic_table_new_name [id=cbbb530872cf4ea7be7e050059219146]) successfully deleted
I20240502 14:00:53.894963 8660 tablet_bootstrap.cc:492] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:53.895045 8660 ts_tablet_manager.cc:1367] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:53.896019 8660 raft_consensus.cc:348] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.896119 8660 raft_consensus.cc:374] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.896154 8660 raft_consensus.cc:729] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.894832 8667 ts_tablet_manager.cc:1909] T 1516806f7adf4f928f27050d7947d9da P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:53.896989 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 1516806f7adf4f928f27050d7947d9da (table rebalancer_dynamic_table_new_name [id=cbbb530872cf4ea7be7e050059219146]) successfully deleted
I20240502 14:00:53.897208 8660 consensus_queue.cc:260] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.898391 8660 ts_tablet_manager.cc:1398] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:53.898707 8673 tablet_bootstrap.cc:492] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:53.898912 7431 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:53.899025 7431 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.899421 7431 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:53.899830 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3fb8ad72148328d8f9d5750c0bd14" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:53.899945 29471 raft_consensus.cc:2454] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:53.900215 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3fb8ad72148328d8f9d5750c0bd14" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:53.900362 31860 raft_consensus.cc:2454] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:53.900375 29537 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:53.900554 7431 raft_consensus.cc:2781] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:53.900601 7431 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:53.900642 7431 raft_consensus.cc:3037] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.901707 7431 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.902062 7431 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:53.902851 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3fb8ad72148328d8f9d5750c0bd14" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:53.902984 31860 raft_consensus.cc:3037] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.903239 8673 tablet_bootstrap.cc:654] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.903956 31860 raft_consensus.cc:2454] T 01b3fb8ad72148328d8f9d5750c0bd14 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:53.904237 29535 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:53.904425 7431 raft_consensus.cc:2781] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:53.904506 7431 raft_consensus.cc:686] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:53.904639 7431 consensus_queue.cc:237] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.904721 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3fb8ad72148328d8f9d5750c0bd14" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:53.904812 29471 raft_consensus.cc:3037] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.905829 29471 raft_consensus.cc:2454] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:53.906484 8673 tablet_bootstrap.cc:492] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:53.906605 8673 ts_tablet_manager.cc:1367] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:53.907056 8673 raft_consensus.cc:348] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.907163 8673 raft_consensus.cc:374] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.907227 8673 raft_consensus.cc:729] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.907155 29327 catalog_manager.cc:5484] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:53.907383 8673 consensus_queue.cc:260] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.907740 8673 ts_tablet_manager.cc:1398] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:53.911235 31992 consensus_queue.cc:237] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:53.912793 32255 raft_consensus.cc:1261] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:53.913156 29600 raft_consensus.cc:1261] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:53.913714 6975 consensus_queue.cc:1035] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:00:53.914053 31927 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1203440714974cb5981e67c2856fb0c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:53.914300 8404 consensus_queue.cc:1035] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:53.915947 8694 raft_consensus.cc:2932] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:53.916359 32254 raft_consensus.cc:2932] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:53.919014 29600 raft_consensus.cc:2932] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:53.919476 29323 catalog_manager.cc:5484] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:53.940753 31992 consensus_queue.cc:237] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.942107 32122 raft_consensus.cc:1261] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:53.942137 31860 raft_consensus.cc:1261] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:53.942267 31926 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d40f25aa6bcc46fca8cd67a6fd4b86b6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:53.942580 8697 consensus_queue.cc:1035] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:53.942764 8694 consensus_queue.cc:1035] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:53.944962 8697 raft_consensus.cc:2932] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:53.948014 32122 raft_consensus.cc:2932] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:53.948014 31860 raft_consensus.cc:2932] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:53.948519 29326 catalog_manager.cc:5484] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:53.962693 8152 raft_consensus.cc:482] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:53.962811 8152 raft_consensus.cc:504] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.963311 8152 leader_election.cc:288] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:53.963939 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfad98900ade46899e78ae1ba7a54dbc" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:53.964062 29864 raft_consensus.cc:2454] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:53.964360 29405 leader_election.cc:302] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:53.964581 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfad98900ade46899e78ae1ba7a54dbc" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:53.964701 29729 raft_consensus.cc:2454] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:53.964780 29323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:53.965376 8152 raft_consensus.cc:2781] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:53.965426 8152 raft_consensus.cc:482] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:53.965459 8152 raft_consensus.cc:3037] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.966537 8152 raft_consensus.cc:504] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.967620 8152 leader_election.cc:288] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:53.968973 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfad98900ade46899e78ae1ba7a54dbc" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:53.969136 29864 raft_consensus.cc:3037] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.970204 29864 raft_consensus.cc:2454] T bfad98900ade46899e78ae1ba7a54dbc P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:53.971822 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfad98900ade46899e78ae1ba7a54dbc" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:53.971944 29732 raft_consensus.cc:3037] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:53.972870 29451 tablet_service.cc:1467] Processing CreateTablet for tablet b777c619d83e40428802f5c8c5c507a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36619b2b56f4576884cfad76ff6cf55]), partition=
I20240502 14:00:53.972872 29582 tablet_service.cc:1467] Processing CreateTablet for tablet b777c619d83e40428802f5c8c5c507a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36619b2b56f4576884cfad76ff6cf55]), partition=
I20240502 14:00:53.972893 29732 raft_consensus.cc:2454] T bfad98900ade46899e78ae1ba7a54dbc P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:53.973101 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b777c619d83e40428802f5c8c5c507a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.973101 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b777c619d83e40428802f5c8c5c507a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.975198 8660 tablet_bootstrap.cc:492] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:53.975260 8659 tablet_bootstrap.cc:492] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:53.975970 29405 leader_election.cc:302] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:53.976094 8660 tablet_bootstrap.cc:654] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.976202 8659 tablet_bootstrap.cc:654] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.976294 8152 raft_consensus.cc:2781] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:53.976380 8152 raft_consensus.cc:686] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:53.976521 8152 consensus_queue.cc:237] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.978117 29843 tablet_service.cc:1467] Processing CreateTablet for tablet b777c619d83e40428802f5c8c5c507a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36619b2b56f4576884cfad76ff6cf55]), partition=
I20240502 14:00:53.978343 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b777c619d83e40428802f5c8c5c507a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:53.979661 29329 catalog_manager.cc:5484] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:53.980504 8673 tablet_bootstrap.cc:492] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:53.980701 8659 tablet_bootstrap.cc:492] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:53.980768 8659 ts_tablet_manager.cc:1367] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 14:00:53.981155 8659 raft_consensus.cc:348] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.981258 8659 raft_consensus.cc:374] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.981298 8659 raft_consensus.cc:729] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.981356 8673 tablet_bootstrap.cc:654] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:53.981426 8659 consensus_queue.cc:260] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.981794 8659 ts_tablet_manager.cc:1398] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:00:53.983992 8673 tablet_bootstrap.cc:492] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:53.984071 8673 ts_tablet_manager.cc:1367] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:53.984455 8673 raft_consensus.cc:348] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.984539 8673 raft_consensus.cc:374] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.984565 8673 raft_consensus.cc:729] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.984668 8673 consensus_queue.cc:260] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.984943 8673 ts_tablet_manager.cc:1398] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:53.986465 8660 tablet_bootstrap.cc:492] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:53.986549 8660 ts_tablet_manager.cc:1367] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:53.987077 8660 raft_consensus.cc:348] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.987192 8660 raft_consensus.cc:374] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:53.987226 8660 raft_consensus.cc:729] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:53.987329 8660 consensus_queue.cc:260] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:53.987593 8660 ts_tablet_manager.cc:1398] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:53.993716 29864 consensus_queue.cc:237] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:53.995862 31860 raft_consensus.cc:1261] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:53.996276 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0001d3f124c42eba6129836209a41fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:53.996450 8067 consensus_queue.cc:1035] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:53.996490 31992 raft_consensus.cc:1261] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:53.996904 8067 consensus_queue.cc:1035] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:53.998665 7551 raft_consensus.cc:2932] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:53.999085 31992 raft_consensus.cc:2932] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.002017 29329 catalog_manager.cc:5484] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.001998 31860 raft_consensus.cc:2932] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.024430 29732 consensus_queue.cc:237] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:54.026023 29864 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:54.026592 8233 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:00:54.026779 29666 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bdf925a9c4d456fb3b4ef448a2154c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.029633 31992 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:54.030910 8233 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:54.032635 8371 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:54.033011 31992 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:54.035837 29864 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:54.039364 29326 catalog_manager.cc:5484] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:54.066140 29600 consensus_queue.cc:237] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.067631 31992 raft_consensus.cc:1261] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.067775 32255 raft_consensus.cc:1261] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:54.067798 29537 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a2ab852e31d4181a8597b9047a08554. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.068280 8776 consensus_queue.cc:1035] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.068652 8311 consensus_queue.cc:1035] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.071647 8311 raft_consensus.cc:2932] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.072067 31992 raft_consensus.cc:2932] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.074571 32254 raft_consensus.cc:2932] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.075518 29329 catalog_manager.cc:5484] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.098217 29732 consensus_queue.cc:237] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:54.099608 31860 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:54.099635 29864 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:00:54.099824 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 463dddd3473a42a68b488e1c465e997f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.100010 8233 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:54.100351 8233 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:54.101965 8233 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.102403 31860 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.104045 29864 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.105700 29326 catalog_manager.cc:5484] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 5 to 6, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.126835 32122 consensus_queue.cc:237] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.128726 31992 raft_consensus.cc:1261] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.128871 29471 raft_consensus.cc:1261] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.129177 8814 consensus_queue.cc:1035] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:00:54.129380 32056 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 541806c11f074316a5342050427f4849. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.129405 7237 consensus_queue.cc:1035] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.131610 8814 raft_consensus.cc:2932] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.132022 29471 raft_consensus.cc:2932] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.134271 31992 raft_consensus.cc:2932] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.135155 29323 catalog_manager.cc:5484] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.141182 8779 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:54.141273 8779 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:54.141651 8779 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:54.141981 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ede8318b945487cae52b2140358b2d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:54.142119 31860 raft_consensus.cc:2454] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:54.142266 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ede8318b945487cae52b2140358b2d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:54.142405 32254 raft_consensus.cc:2454] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 0.
I20240502 14:00:54.142418 29535 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:54.142573 8779 raft_consensus.cc:2781] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:54.142629 8779 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:54.142665 8779 raft_consensus.cc:3037] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.143680 8779 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:54.143785 8152 raft_consensus.cc:482] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:54.143899 8152 raft_consensus.cc:504] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.144022 8779 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:54.144243 8152 leader_election.cc:288] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:54.144315 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ede8318b945487cae52b2140358b2d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:54.144436 31860 raft_consensus.cc:3037] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.144712 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ede8318b945487cae52b2140358b2d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:54.144816 32254 raft_consensus.cc:3037] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.145308 31860 raft_consensus.cc:2454] T 7ede8318b945487cae52b2140358b2d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:54.145601 29535 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:54.145793 32254 raft_consensus.cc:2454] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:00:54.145799 8779 raft_consensus.cc:2781] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:54.145891 8779 raft_consensus.cc:686] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:00:54.146044 8779 consensus_queue.cc:237] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:54.146798 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b777c619d83e40428802f5c8c5c507a9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:54.146883 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b777c619d83e40428802f5c8c5c507a9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:54.146972 29600 raft_consensus.cc:2454] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:54.146973 29864 raft_consensus.cc:2454] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:54.147222 29405 leader_election.cc:302] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:54.147512 8152 raft_consensus.cc:2781] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:54.147562 8152 raft_consensus.cc:482] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:54.147599 8152 raft_consensus.cc:3037] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.148506 8152 raft_consensus.cc:504] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.148823 8152 leader_election.cc:288] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:54.150105 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b777c619d83e40428802f5c8c5c507a9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:54.150215 29864 raft_consensus.cc:3037] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.150238 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b777c619d83e40428802f5c8c5c507a9" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:54.150324 29600 raft_consensus.cc:3037] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.150561 29326 catalog_manager.cc:5484] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:54.151186 29864 raft_consensus.cc:2454] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:54.151283 29600 raft_consensus.cc:2454] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:54.151564 29404 leader_election.cc:302] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:00:54.153713 8152 raft_consensus.cc:2781] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:54.153803 8152 raft_consensus.cc:686] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:54.153923 8152 consensus_queue.cc:237] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.156430 29326 catalog_manager.cc:5484] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:54.168120 32254 consensus_queue.cc:237] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:54.169847 31860 raft_consensus.cc:1261] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.169886 32122 raft_consensus.cc:1261] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:54.170066 32189 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39db58bc129046bbbefea02b3eee4243. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.170627 7188 consensus_queue.cc:1035] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.170817 8071 consensus_queue.cc:1035] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.172812 7188 raft_consensus.cc:2932] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.175434 31860 raft_consensus.cc:2932] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.176755 29326 catalog_manager.cc:5484] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.179260 32122 raft_consensus.cc:2932] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.198928 31990 consensus_queue.cc:237] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.200289 29732 raft_consensus.cc:1261] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.200346 32254 raft_consensus.cc:1261] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:54.200565 31927 consensus_peers.cc:485] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 824fb738a4b84e13b5fc283a26c92355. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.200747 8697 consensus_queue.cc:1035] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.200942 8694 consensus_queue.cc:1035] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.202632 8694 raft_consensus.cc:2932] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.203241 29732 raft_consensus.cc:2932] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.204970 32254 raft_consensus.cc:2932] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.207692 29327 catalog_manager.cc:5484] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.229099 31990 consensus_queue.cc:237] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:54.230654 29864 raft_consensus.cc:1261] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.230968 29729 raft_consensus.cc:1261] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.231284 8404 consensus_queue.cc:1035] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:00:54.231592 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ec7b45e60d140d4a6e937307370fdc9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.231894 8404 consensus_queue.cc:1035] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.233758 8404 raft_consensus.cc:2932] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.234150 29729 raft_consensus.cc:2932] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.235694 29864 raft_consensus.cc:2932] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.237565 29326 catalog_manager.cc:5484] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:54.260475 32122 consensus_queue.cc:237] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.261857 31990 raft_consensus.cc:1261] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:54.261907 32056 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75347e9c867547efaf78355aec6dfb40. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.261952 29864 raft_consensus.cc:1261] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.262285 7237 consensus_queue.cc:1035] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.262463 8814 consensus_queue.cc:1035] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.264549 7237 raft_consensus.cc:2932] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.264998 29864 raft_consensus.cc:2932] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.266717 31990 raft_consensus.cc:2932] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.268833 29327 catalog_manager.cc:5484] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.287750 32122 consensus_queue.cc:237] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
W20240502 14:00:54.289304 32057 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8b939db0e5b464dbcecbf8e18cccadf. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.289397 29471 raft_consensus.cc:1261] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.289824 7237 consensus_queue.cc:1035] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.289969 31859 raft_consensus.cc:1261] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.290397 7237 consensus_queue.cc:1035] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.290470 29327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:54.293159 7237 raft_consensus.cc:2932] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.295104 31859 raft_consensus.cc:2932] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.295516 29470 raft_consensus.cc:2932] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.296351 29864 raft_consensus.cc:1261] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.296797 8152 consensus_queue.cc:1035] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.297276 29600 raft_consensus.cc:1261] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.297796 8655 consensus_queue.cc:1035] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.298557 29327 catalog_manager.cc:5484] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:54.299871 8914 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:54.300218 8914 tablet.cc:1620] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:54.300556 8915 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:54.300945 8915 tablet.cc:1620] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:54.303193 29323 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet b777c619d83e40428802f5c8c5c507a9 (table rebalancer_dynamic_table [id=a36619b2b56f4576884cfad76ff6cf55]). Expected version 1 got 0
I20240502 14:00:54.305759 29329 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=a36619b2b56f4576884cfad76ff6cf55] alter complete (version 1)
I20240502 14:00:54.307580 29329 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet b777c619d83e40428802f5c8c5c507a9 (table rebalancer_dynamic_table [id=a36619b2b56f4576884cfad76ff6cf55]). Expected version 1 got 0
I20240502 14:00:54.308400 8916 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:54.308790 8916 tablet.cc:1620] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:54.333856 31859 consensus_queue.cc:237] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
W20240502 14:00:54.335486 31796 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 631725c869674c45ac79c3f05cee8356. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.335556 29864 raft_consensus.cc:1261] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.335598 32254 raft_consensus.cc:1261] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.335987 8859 consensus_queue.cc:1035] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.336151 6614 consensus_queue.cc:1035] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.344214 8859 raft_consensus.cc:2932] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.345170 29864 raft_consensus.cc:2932] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.347159 32254 raft_consensus.cc:2932] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.348917 29327 catalog_manager.cc:5484] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:54.375309 29864 consensus_queue.cc:237] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:54.376767 32122 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:54.377108 29729 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:54.377238 8067 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:00:54.377420 29797 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be8afc0891fb43fa9ffc863ad2bc90d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.377648 8067 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:54.379241 8739 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.379648 29732 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.382143 32122 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.382866 29329 catalog_manager.cc:5484] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.404522 32254 consensus_queue.cc:237] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.406477 32122 raft_consensus.cc:1261] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:54.406888 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ea3465ad91540cea8906bcc406efd07. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.407090 8656 consensus_queue.cc:1035] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.407544 31990 raft_consensus.cc:1261] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.407946 8656 consensus_queue.cc:1035] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.409677 8656 raft_consensus.cc:2932] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.409989 31990 raft_consensus.cc:2932] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.410145 8776 consensus_queue.cc:1035] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.411999 32122 raft_consensus.cc:2932] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.413401 29326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:54.415529 29329 catalog_manager.cc:5484] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.421317 8914 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:54.421679 8914 tablet.cc:1620] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:54.422338 8915 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:54.422631 8915 tablet.cc:1620] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:54.430418 8779 consensus_queue.cc:1035] T 01b3fb8ad72148328d8f9d5750c0bd14 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.431883 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=a36619b2b56f4576884cfad76ff6cf55] alter complete (version 2)
I20240502 14:00:54.434271 8984 ts_tablet_manager.cc:902] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:54.436789 8916 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:54.437126 8916 tablet.cc:1620] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:54.443048 8984 tablet_copy_client.cc:308] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:54.443763 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 9bdf925a9c4d456fb3b4ef448a2154c6 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:57583)
I20240502 14:00:54.443858 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 9bdf925a9c4d456fb3b4ef448a2154c6 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:57583: session id = f0e01c4f581843e9ad4ced97f0072dc4-9bdf925a9c4d456fb3b4ef448a2154c6
I20240502 14:00:54.444963 29753 tablet_copy_source_session.cc:205] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.446498 8984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdf925a9c4d456fb3b4ef448a2154c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.449020 8984 tablet_copy_client.cc:783] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.449172 8984 tablet_copy_client.cc:647] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.450230 8984 tablet_copy_client.cc:523] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.451462 8984 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:54.452965 32254 consensus_queue.cc:237] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.454253 31859 raft_consensus.cc:1261] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.454391 29732 raft_consensus.cc:1261] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:54.454500 32189 consensus_peers.cc:485] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ae0eeb814e34729b6284d838d458ddf. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.454749 8071 consensus_queue.cc:1035] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.454933 8656 consensus_queue.cc:1035] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.458431 8999 ts_tablet_manager.cc:902] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:54.458989 8999 tablet_copy_client.cc:308] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:54.459800 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:44929)
I20240502 14:00:54.459900 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:44929: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-d40f25aa6bcc46fca8cd67a6fd4b86b6
I20240502 14:00:54.459882 8656 raft_consensus.cc:2932] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:54.460629 32012 tablet_copy_source_session.cc:205] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.462599 29732 raft_consensus.cc:2932] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:54.464744 31859 raft_consensus.cc:2932] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:54.465009 29323 catalog_manager.cc:5484] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:54.467272 8999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.471210 8999 tablet_copy_client.cc:783] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.471431 8999 tablet_copy_client.cc:647] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.474498 8999 tablet_copy_client.cc:523] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.475730 8999 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:54.479108 8999 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.479375 8999 tablet_bootstrap.cc:492] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:00:54.479609 8999 ts_tablet_manager.cc:1367] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:00:54.480150 8999 raft_consensus.cc:348] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.480360 8999 raft_consensus.cc:729] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.480535 8999 consensus_queue.cc:260] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.481410 8999 ts_tablet_manager.cc:1398] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:54.481412 8984 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.481715 8984 tablet_bootstrap.cc:492] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:00:54.481829 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-d40f25aa6bcc46fca8cd67a6fd4b86b6 received from {username='slave'} at 127.22.55.132:44929
I20240502 14:00:54.481889 8984 ts_tablet_manager.cc:1367] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.002s
I20240502 14:00:54.481902 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-d40f25aa6bcc46fca8cd67a6fd4b86b6 on tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:54.482394 8984 raft_consensus.cc:348] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:54.482568 8984 raft_consensus.cc:729] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.482751 8984 consensus_queue.cc:260] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:54.483026 8984 ts_tablet_manager.cc:1398] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.484126 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-9bdf925a9c4d456fb3b4ef448a2154c6 received from {username='slave'} at 127.22.55.133:57583
I20240502 14:00:54.484212 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-9bdf925a9c4d456fb3b4ef448a2154c6 on tablet 9bdf925a9c4d456fb3b4ef448a2154c6 with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:00:54.486671 9012 ts_tablet_manager.cc:902] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:00:54.487262 9012 tablet_copy_client.cc:308] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:00:54.488020 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 463dddd3473a42a68b488e1c465e997f from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:55687)
I20240502 14:00:54.488126 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 463dddd3473a42a68b488e1c465e997f from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:55687: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-463dddd3473a42a68b488e1c465e997f
I20240502 14:00:54.489061 29753 tablet_copy_source_session.cc:205] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.489898 9012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463dddd3473a42a68b488e1c465e997f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.494761 9012 tablet_copy_client.cc:783] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.494939 9012 tablet_copy_client.cc:647] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.496253 9012 tablet_copy_client.cc:523] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.497529 9012 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:54.503417 29470 consensus_queue.cc:237] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:54.504696 29732 raft_consensus.cc:1261] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.504936 31859 raft_consensus.cc:1261] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.505347 8448 consensus_queue.cc:1035] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:00:54.505628 29406 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): Couldn't send request to peer 8973f2527acb48f5ad36589652de73e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fad144c461044c299ea905832908666a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.505730 8655 consensus_queue.cc:1035] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.512720 8655 raft_consensus.cc:2932] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:54.514891 31859 raft_consensus.cc:2932] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:54.516654 29327 catalog_manager.cc:5484] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:54.518707 8152 consensus_queue.cc:1035] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.521351 29732 raft_consensus.cc:2932] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:54.530885 9036 ts_tablet_manager.cc:902] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:54.531499 9036 tablet_copy_client.cc:308] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:54.532339 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet 1203440714974cb5981e67c2856fb0c7 from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:56031)
I20240502 14:00:54.532464 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet 1203440714974cb5981e67c2856fb0c7 from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:56031: session id = 1d00d3a47c52478894cf8c66959ef7bd-1203440714974cb5981e67c2856fb0c7
I20240502 14:00:54.533378 32012 tablet_copy_source_session.cc:205] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.535123 8655 consensus_queue.cc:1035] T bfad98900ade46899e78ae1ba7a54dbc P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.535815 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1203440714974cb5981e67c2856fb0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.536096 9012 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.537109 9012 tablet_bootstrap.cc:492] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:54.537287 9012 ts_tablet_manager.cc:1367] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.004s
I20240502 14:00:54.537724 9012 raft_consensus.cc:348] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:54.537832 9012 raft_consensus.cc:729] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.537953 9036 tablet_copy_client.cc:783] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.538111 9036 tablet_copy_client.cc:647] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.538275 9012 consensus_queue.cc:260] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:54.538684 9012 ts_tablet_manager.cc:1398] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.539096 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:54.539831 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-463dddd3473a42a68b488e1c465e997f received from {username='slave'} at 127.22.55.135:55687
I20240502 14:00:54.539911 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-463dddd3473a42a68b488e1c465e997f on tablet 463dddd3473a42a68b488e1c465e997f with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:54.545035 9036 tablet_copy_client.cc:523] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.547724 8914 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:54.548806 9036 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:54.549396 8916 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:54.550055 8915 tablet.cc:1722] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:54.555054 29327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=a36619b2b56f4576884cfad76ff6cf55] alter complete (version 3)
I20240502 14:00:54.557129 9036 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.557371 9036 tablet_bootstrap.cc:492] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:00:54.557497 9036 ts_tablet_manager.cc:1367] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:00:54.557847 9036 raft_consensus.cc:348] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.557940 9036 raft_consensus.cc:729] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.558030 9036 consensus_queue.cc:260] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.558266 9036 ts_tablet_manager.cc:1398] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:54.559352 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-1203440714974cb5981e67c2856fb0c7 received from {username='slave'} at 127.22.55.129:56031
I20240502 14:00:54.559438 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-1203440714974cb5981e67c2856fb0c7 on tablet 1203440714974cb5981e67c2856fb0c7 with peer 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:54.567030 9050 ts_tablet_manager.cc:902] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:54.568635 9050 tablet_copy_client.cc:308] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:54.569078 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet a0001d3f124c42eba6129836209a41fa from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:51467)
I20240502 14:00:54.569196 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet a0001d3f124c42eba6129836209a41fa from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:51467: session id = d6e2e2c10def4331bec17bd110c9b423-a0001d3f124c42eba6129836209a41fa
I20240502 14:00:54.570016 29884 tablet_copy_source_session.cc:205] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.570837 9050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0001d3f124c42eba6129836209a41fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.572747 31859 consensus_queue.cc:237] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:54.573391 9050 tablet_copy_client.cc:783] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.573585 9050 tablet_copy_client.cc:647] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.574774 29470 raft_consensus.cc:1261] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.574868 32254 raft_consensus.cc:1261] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:54.575245 31794 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3afd58ca519e4bc8a6ab00c37abf9f1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.575452 6614 consensus_queue.cc:1035] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.575748 9050 tablet_copy_client.cc:523] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.575778 6614 consensus_queue.cc:1035] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.577092 9050 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:54.577684 7235 raft_consensus.cc:2932] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.578230 29470 raft_consensus.cc:2932] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.580583 9050 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.580816 9050 tablet_bootstrap.cc:492] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:54.580960 9050 ts_tablet_manager.cc:1367] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 14:00:54.581413 9050 raft_consensus.cc:348] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.581522 9050 raft_consensus.cc:729] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.581651 9050 consensus_queue.cc:260] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.581809 29323 catalog_manager.cc:5484] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.581960 9050 ts_tablet_manager.cc:1398] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:54.582607 32254 raft_consensus.cc:2932] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:54.585968 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-a0001d3f124c42eba6129836209a41fa received from {username='slave'} at 127.22.55.130:51467
I20240502 14:00:54.586053 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-a0001d3f124c42eba6129836209a41fa on tablet a0001d3f124c42eba6129836209a41fa with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:54.593915 9075 ts_tablet_manager.cc:902] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:54.594482 9075 tablet_copy_client.cc:308] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:00:54.594954 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet 39db58bc129046bbbefea02b3eee4243 from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:39103)
I20240502 14:00:54.595041 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet 39db58bc129046bbbefea02b3eee4243 from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:39103: session id = efdbf7bdfe0b41bdadb1add053b09a7e-39db58bc129046bbbefea02b3eee4243
I20240502 14:00:54.596100 32275 tablet_copy_source_session.cc:205] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.597083 9075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39db58bc129046bbbefea02b3eee4243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.599202 9075 tablet_copy_client.cc:783] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.599361 9075 tablet_copy_client.cc:647] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.600260 9075 tablet_copy_client.cc:523] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.601444 9075 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:54.603595 31992 consensus_queue.cc:237] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.604765 9075 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.605064 9075 tablet_bootstrap.cc:492] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:00:54.605289 9075 ts_tablet_manager.cc:1367] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240502 14:00:54.605672 31859 raft_consensus.cc:1261] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.605778 9075 raft_consensus.cc:348] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:54.605876 9075 raft_consensus.cc:729] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.606019 32122 raft_consensus.cc:1261] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:54.606007 9075 consensus_queue.cc:260] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:54.606047 8694 consensus_queue.cc:1035] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.606331 9075 ts_tablet_manager.cc:1398] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 14:00:54.606676 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 768df43846d94951a3a48f6f626bb473. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:54.606868 8694 consensus_queue.cc:1035] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:54.607348 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-39db58bc129046bbbefea02b3eee4243 received from {username='slave'} at 127.22.55.131:39103
I20240502 14:00:54.607424 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-39db58bc129046bbbefea02b3eee4243 on tablet 39db58bc129046bbbefea02b3eee4243 with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:54.608853 8776 consensus_queue.cc:1035] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.611101 8694 raft_consensus.cc:2932] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.612972 8779 consensus_queue.cc:1035] T 7ede8318b945487cae52b2140358b2d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:54.613780 32122 raft_consensus.cc:2932] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.614204 29323 catalog_manager.cc:5484] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.615816 31861 raft_consensus.cc:2932] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.631744 9012 ts_tablet_manager.cc:902] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:54.632201 9012 tablet_copy_client.cc:308] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:54.632660 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet 0ec7b45e60d140d4a6e937307370fdc9 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb ({username='slave'} at 127.22.55.135:38501)
I20240502 14:00:54.632761 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet 0ec7b45e60d140d4a6e937307370fdc9 from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb at {username='slave'} at 127.22.55.135:38501: session id = f6b3fa3988ae4af9ba8cc5ffb79d5bfb-0ec7b45e60d140d4a6e937307370fdc9
I20240502 14:00:54.633797 32012 tablet_copy_source_session.cc:205] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.634922 9012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7b45e60d140d4a6e937307370fdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.637171 9012 tablet_copy_client.cc:783] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.637308 9012 tablet_copy_client.cc:647] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.638307 9012 tablet_copy_client.cc:523] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.639564 9012 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:54.643189 9012 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.643417 9012 tablet_bootstrap.cc:492] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap complete.
I20240502 14:00:54.643574 9012 ts_tablet_manager.cc:1367] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 14:00:54.643929 9012 raft_consensus.cc:348] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:54.644029 9012 raft_consensus.cc:729] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.644189 9012 consensus_queue.cc:260] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:54.644613 9012 ts_tablet_manager.cc:1398] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.645212 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-0ec7b45e60d140d4a6e937307370fdc9 received from {username='slave'} at 127.22.55.135:38501
I20240502 14:00:54.645283 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session f6b3fa3988ae4af9ba8cc5ffb79d5bfb-0ec7b45e60d140d4a6e937307370fdc9 on tablet 0ec7b45e60d140d4a6e937307370fdc9 with peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:00:54.657168 29327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:54.657302 29327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:54.659456 29327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet b777c619d83e40428802f5c8c5c507a9
I20240502 14:00:54.659993 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet b777c619d83e40428802f5c8c5c507a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:54 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:54.660055 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet b777c619d83e40428802f5c8c5c507a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:54 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:54.660147 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet b777c619d83e40428802f5c8c5c507a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:54 UTC) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:54.660310 9118 tablet_replica.cc:330] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:54.660533 9120 tablet_replica.cc:330] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:54.660523 9118 raft_consensus.cc:2229] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:54.660559 9119 tablet_replica.cc:330] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:54.660689 9119 raft_consensus.cc:2229] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:54.660813 9118 raft_consensus.cc:2258] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:54.661041 9120 raft_consensus.cc:2229] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:54.661058 9119 raft_consensus.cc:2258] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:54.661293 9120 raft_consensus.cc:2258] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:54.661499 9118 ts_tablet_manager.cc:1875] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:54.661700 9119 ts_tablet_manager.cc:1875] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:54.663578 9120 ts_tablet_manager.cc:1875] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:54.664476 9118 ts_tablet_manager.cc:1888] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:54.664597 9118 log.cc:1196] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/b777c619d83e40428802f5c8c5c507a9
I20240502 14:00:54.665028 9118 ts_tablet_manager.cc:1909] T b777c619d83e40428802f5c8c5c507a9 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:54.665472 9119 ts_tablet_manager.cc:1888] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:54.665647 9119 log.cc:1196] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/b777c619d83e40428802f5c8c5c507a9
I20240502 14:00:54.665848 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet b777c619d83e40428802f5c8c5c507a9 (table rebalancer_dynamic_table_new_name [id=a36619b2b56f4576884cfad76ff6cf55]) successfully deleted
I20240502 14:00:54.666100 9119 ts_tablet_manager.cc:1909] T b777c619d83e40428802f5c8c5c507a9 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:54.666906 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet b777c619d83e40428802f5c8c5c507a9 (table rebalancer_dynamic_table_new_name [id=a36619b2b56f4576884cfad76ff6cf55]) successfully deleted
I20240502 14:00:54.668048 9120 ts_tablet_manager.cc:1888] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:54.668182 9120 log.cc:1196] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/b777c619d83e40428802f5c8c5c507a9
I20240502 14:00:54.668581 9120 ts_tablet_manager.cc:1909] T b777c619d83e40428802f5c8c5c507a9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting consensus metadata
I20240502 14:00:54.669278 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet b777c619d83e40428802f5c8c5c507a9 (table rebalancer_dynamic_table_new_name [id=a36619b2b56f4576884cfad76ff6cf55]) successfully deleted
I20240502 14:00:54.676841 9036 ts_tablet_manager.cc:902] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:54.677316 9036 tablet_copy_client.cc:308] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:00:54.677723 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 4a2ab852e31d4181a8597b9047a08554 from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:52275)
I20240502 14:00:54.677814 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 4a2ab852e31d4181a8597b9047a08554 from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:52275: session id = 1d00d3a47c52478894cf8c66959ef7bd-4a2ab852e31d4181a8597b9047a08554
I20240502 14:00:54.678740 29622 tablet_copy_source_session.cc:205] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.679587 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2ab852e31d4181a8597b9047a08554. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.681552 9036 tablet_copy_client.cc:783] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.681675 9036 tablet_copy_client.cc:647] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.682435 29325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:54.686898 9036 tablet_copy_client.cc:523] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.688102 9036 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:54.688822 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 266aa0d4f078466a820bd48e60d044aa (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.689098 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266aa0d4f078466a820bd48e60d044aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.689342 31971 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2dc222951548708862c3adf205d0ef (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.689420 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 266aa0d4f078466a820bd48e60d044aa (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.689623 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2dc222951548708862c3adf205d0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.689666 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266aa0d4f078466a820bd48e60d044aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.691928 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2dc222951548708862c3adf205d0ef (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.692059 31970 tablet_service.cc:1467] Processing CreateTablet for tablet 708b6a4007e1486ba7269d23962891ac (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.692190 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2dc222951548708862c3adf205d0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.692255 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708b6a4007e1486ba7269d23962891ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.694684 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 266aa0d4f078466a820bd48e60d044aa (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.694958 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266aa0d4f078466a820bd48e60d044aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.697219 9149 tablet_bootstrap.cc:492] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:54.698066 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 708b6a4007e1486ba7269d23962891ac (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.698154 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2dc222951548708862c3adf205d0ef (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.698338 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708b6a4007e1486ba7269d23962891ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.698498 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2dc222951548708862c3adf205d0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.698786 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 708b6a4007e1486ba7269d23962891ac (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=102a1aeae64d491a96ac9276d2a15ec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:54.699051 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708b6a4007e1486ba7269d23962891ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.700908 9148 tablet_bootstrap.cc:492] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:54.702056 9145 tablet_bootstrap.cc:492] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:54.703333 9148 tablet_bootstrap.cc:654] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.703758 9149 tablet_bootstrap.cc:654] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.705335 9145 tablet_bootstrap.cc:654] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.706018 9152 tablet_bootstrap.cc:492] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:54.707007 9152 tablet_bootstrap.cc:654] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.707747 9153 tablet_bootstrap.cc:492] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:54.708408 9148 tablet_bootstrap.cc:492] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:54.708537 9148 ts_tablet_manager.cc:1367] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:54.708703 9153 tablet_bootstrap.cc:654] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.708968 9148 raft_consensus.cc:348] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.709205 9148 raft_consensus.cc:374] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.709281 9148 raft_consensus.cc:729] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.709481 9148 consensus_queue.cc:260] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.709527 9050 ts_tablet_manager.cc:902] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:54.710098 9050 tablet_copy_client.cc:308] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:54.711274 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 541806c11f074316a5342050427f4849 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:35209)
I20240502 14:00:54.711359 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 541806c11f074316a5342050427f4849 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:35209: session id = d6e2e2c10def4331bec17bd110c9b423-541806c11f074316a5342050427f4849
I20240502 14:00:54.712172 32143 tablet_copy_source_session.cc:205] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.714411 9050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541806c11f074316a5342050427f4849. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.716485 9050 tablet_copy_client.cc:783] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.716616 9050 tablet_copy_client.cc:647] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.716923 9147 tablet_bootstrap.cc:492] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:54.718227 9036 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.718482 9036 tablet_bootstrap.cc:492] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:00:54.718632 9036 ts_tablet_manager.cc:1367] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240502 14:00:54.718758 9147 tablet_bootstrap.cc:654] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.720180 9036 raft_consensus.cc:348] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.720295 9036 raft_consensus.cc:729] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.720425 9036 consensus_queue.cc:260] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.720721 9036 ts_tablet_manager.cc:1398] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:54.721028 9148 ts_tablet_manager.cc:1398] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 14:00:54.721724 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-4a2ab852e31d4181a8597b9047a08554 received from {username='slave'} at 127.22.55.129:52275
I20240502 14:00:54.721805 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-4a2ab852e31d4181a8597b9047a08554 on tablet 4a2ab852e31d4181a8597b9047a08554 with peer 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:54.725107 9146 tablet_bootstrap.cc:492] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:54.726640 9146 tablet_bootstrap.cc:654] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.727478 9152 tablet_bootstrap.cc:492] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:54.727623 9152 ts_tablet_manager.cc:1367] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240502 14:00:54.728152 9152 raft_consensus.cc:348] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.728338 9152 raft_consensus.cc:374] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.728423 9152 raft_consensus.cc:729] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.728585 9152 consensus_queue.cc:260] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.728951 9152 ts_tablet_manager.cc:1398] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.730072 9152 tablet_bootstrap.cc:492] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:54.732259 9182 ts_tablet_manager.cc:902] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:54.732702 9182 tablet_copy_client.cc:308] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:54.733263 9152 tablet_bootstrap.cc:654] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.733326 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet 824fb738a4b84e13b5fc283a26c92355 from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:56031)
I20240502 14:00:54.733426 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet 824fb738a4b84e13b5fc283a26c92355 from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:56031: session id = 1d00d3a47c52478894cf8c66959ef7bd-824fb738a4b84e13b5fc283a26c92355
I20240502 14:00:54.734431 32012 tablet_copy_source_session.cc:205] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.734679 9036 ts_tablet_manager.cc:902] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:54.735121 9036 tablet_copy_client.cc:308] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:00:54.735857 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet 631725c869674c45ac79c3f05cee8356 from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:44365)
I20240502 14:00:54.735951 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet 631725c869674c45ac79c3f05cee8356 from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:44365: session id = 1d00d3a47c52478894cf8c66959ef7bd-631725c869674c45ac79c3f05cee8356
I20240502 14:00:54.736882 31865 tablet_copy_source_session.cc:205] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.737735 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631725c869674c45ac79c3f05cee8356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.739369 9149 tablet_bootstrap.cc:492] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:54.739516 9149 ts_tablet_manager.cc:1367] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240502 14:00:54.741254 9036 tablet_copy_client.cc:783] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.741400 9036 tablet_copy_client.cc:647] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.741631 9149 raft_consensus.cc:348] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.741789 9149 raft_consensus.cc:374] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.741853 9149 raft_consensus.cc:729] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.742157 9149 consensus_queue.cc:260] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.742467 9149 ts_tablet_manager.cc:1398] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:54.742722 9050 tablet_copy_client.cc:523] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.743999 9145 tablet_bootstrap.cc:492] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:54.744086 9145 ts_tablet_manager.cc:1367] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240502 14:00:54.744201 9050 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:54.744551 9145 raft_consensus.cc:348] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.744657 9145 raft_consensus.cc:374] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.744712 9145 raft_consensus.cc:729] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.744833 9145 consensus_queue.cc:260] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.745150 9145 ts_tablet_manager.cc:1398] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.748822 9152 tablet_bootstrap.cc:492] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:54.748958 9152 ts_tablet_manager.cc:1367] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:00:54.749888 9152 raft_consensus.cc:348] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.750051 9152 raft_consensus.cc:374] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.750135 9152 raft_consensus.cc:729] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.750348 9152 consensus_queue.cc:260] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.752267 9152 ts_tablet_manager.cc:1398] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:54.753674 9146 tablet_bootstrap.cc:492] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:54.753829 9146 ts_tablet_manager.cc:1367] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 14:00:54.755342 9146 raft_consensus.cc:348] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.755524 9146 raft_consensus.cc:374] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.755610 9146 raft_consensus.cc:729] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.755780 9146 consensus_queue.cc:260] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.757504 9146 ts_tablet_manager.cc:1398] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:54.757845 9146 tablet_bootstrap.cc:492] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:54.761520 9146 tablet_bootstrap.cc:654] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.762254 29328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:54.762729 9195 ts_tablet_manager.cc:902] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:54.763334 9195 tablet_copy_client.cc:308] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:54.763592 9036 tablet_copy_client.cc:523] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.764039 9147 tablet_bootstrap.cc:492] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:54.764170 9147 ts_tablet_manager.cc:1367] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.001s
I20240502 14:00:54.764783 9147 raft_consensus.cc:348] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.764963 9147 raft_consensus.cc:374] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.765043 9147 raft_consensus.cc:729] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.765105 9036 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:54.765558 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 75347e9c867547efaf78355aec6dfb40 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:35209)
I20240502 14:00:54.765496 9147 consensus_queue.cc:260] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.765655 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 75347e9c867547efaf78355aec6dfb40 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:35209: session id = d6e2e2c10def4331bec17bd110c9b423-75347e9c867547efaf78355aec6dfb40
I20240502 14:00:54.766150 9147 ts_tablet_manager.cc:1398] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:54.766574 32143 tablet_copy_source_session.cc:205] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.768195 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75347e9c867547efaf78355aec6dfb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.769150 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 370d7f219b594aef92b19cb1a8819ba5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295]), partition=
I20240502 14:00:54.769397 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d7f219b594aef92b19cb1a8819ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.770671 9050 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.770964 9050 tablet_bootstrap.cc:492] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:54.771121 9050 ts_tablet_manager.cc:1367] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s
I20240502 14:00:54.771574 9050 raft_consensus.cc:348] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.771688 9050 raft_consensus.cc:729] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.771876 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 370d7f219b594aef92b19cb1a8819ba5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295]), partition=
I20240502 14:00:54.771837 9050 consensus_queue.cc:260] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.772056 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d7f219b594aef92b19cb1a8819ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.772127 9050 ts_tablet_manager.cc:1398] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.772514 9153 tablet_bootstrap.cc:492] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:54.772502 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 370d7f219b594aef92b19cb1a8819ba5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295]), partition=
I20240502 14:00:54.772662 9153 ts_tablet_manager.cc:1367] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.000s
I20240502 14:00:54.772711 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d7f219b594aef92b19cb1a8819ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.773125 9153 raft_consensus.cc:348] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.773217 9153 raft_consensus.cc:374] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.773243 9153 raft_consensus.cc:729] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.773357 9153 consensus_queue.cc:260] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.774379 9153 ts_tablet_manager.cc:1398] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:54.775633 9148 tablet_bootstrap.cc:492] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:54.776510 9148 tablet_bootstrap.cc:654] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.777805 9149 tablet_bootstrap.cc:492] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:54.778700 9149 tablet_bootstrap.cc:654] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.781049 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-541806c11f074316a5342050427f4849 received from {username='slave'} at 127.22.55.130:35209
I20240502 14:00:54.781144 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-541806c11f074316a5342050427f4849 on tablet 541806c11f074316a5342050427f4849 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:54.781252 9145 tablet_bootstrap.cc:492] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:54.782292 9145 tablet_bootstrap.cc:654] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:54.790769 9195 tablet_copy_client.cc:783] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.791028 9195 tablet_copy_client.cc:647] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.792428 9148 tablet_bootstrap.cc:492] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:54.792512 9148 ts_tablet_manager.cc:1367] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 14:00:54.792938 9148 raft_consensus.cc:348] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.793032 9148 raft_consensus.cc:374] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.793058 9148 raft_consensus.cc:729] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.793195 9148 consensus_queue.cc:260] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.793450 9148 ts_tablet_manager.cc:1398] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:54.794040 9149 tablet_bootstrap.cc:492] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:54.794148 9149 ts_tablet_manager.cc:1367] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:00:54.794651 9149 raft_consensus.cc:348] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.794754 9149 raft_consensus.cc:374] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.794786 9149 raft_consensus.cc:729] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.794937 9149 consensus_queue.cc:260] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.795228 9149 ts_tablet_manager.cc:1398] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.795753 9036 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.796010 9036 tablet_bootstrap.cc:492] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:00:54.796175 9036 ts_tablet_manager.cc:1367] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.003s
I20240502 14:00:54.796598 9036 raft_consensus.cc:348] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.796713 9036 raft_consensus.cc:729] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.796831 9036 consensus_queue.cc:260] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.797139 9036 ts_tablet_manager.cc:1398] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:54.798563 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-631725c869674c45ac79c3f05cee8356 received from {username='slave'} at 127.22.55.129:44365
I20240502 14:00:54.798595 9182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824fb738a4b84e13b5fc283a26c92355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.798650 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-631725c869674c45ac79c3f05cee8356 on tablet 631725c869674c45ac79c3f05cee8356 with peer 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:54.799139 9145 tablet_bootstrap.cc:492] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:54.799239 9145 ts_tablet_manager.cc:1367] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:00:54.799736 9145 raft_consensus.cc:348] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.799839 9145 raft_consensus.cc:374] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.799875 9145 raft_consensus.cc:729] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.799999 9145 consensus_queue.cc:260] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.800279 9145 ts_tablet_manager.cc:1398] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.800863 9182 tablet_copy_client.cc:783] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.800994 9182 tablet_copy_client.cc:647] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.811167 9182 tablet_copy_client.cc:523] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.812222 9146 tablet_bootstrap.cc:492] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:54.812328 9146 ts_tablet_manager.cc:1367] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20240502 14:00:54.812742 9146 raft_consensus.cc:348] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.812837 9146 raft_consensus.cc:374] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:54.812863 9146 raft_consensus.cc:729] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.812961 9146 consensus_queue.cc:260] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.813220 9146 ts_tablet_manager.cc:1398] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:54.814960 9182 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:54.821552 9195 tablet_copy_client.cc:523] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.822976 9195 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:54.826799 9195 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.827600 9195 tablet_bootstrap.cc:492] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:54.827785 9195 ts_tablet_manager.cc:1367] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:00:54.828205 9195 raft_consensus.cc:348] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.828316 9195 raft_consensus.cc:729] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.828413 9195 consensus_queue.cc:260] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.828644 9195 ts_tablet_manager.cc:1398] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.829447 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-75347e9c867547efaf78355aec6dfb40 received from {username='slave'} at 127.22.55.130:35209
I20240502 14:00:54.829521 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-75347e9c867547efaf78355aec6dfb40 on tablet 75347e9c867547efaf78355aec6dfb40 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:54.833670 9182 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.833992 9182 tablet_bootstrap.cc:492] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:00:54.836344 9182 ts_tablet_manager.cc:1367] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240502 14:00:54.836927 9182 raft_consensus.cc:348] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.837136 9182 raft_consensus.cc:729] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.837299 9182 consensus_queue.cc:260] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:54.837841 9182 ts_tablet_manager.cc:1398] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:54.838783 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-824fb738a4b84e13b5fc283a26c92355 received from {username='slave'} at 127.22.55.129:56031
I20240502 14:00:54.838881 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-824fb738a4b84e13b5fc283a26c92355 on tablet 824fb738a4b84e13b5fc283a26c92355 with peer 1d00d3a47c52478894cf8c66959ef7bd
I20240502 14:00:54.849179 9075 ts_tablet_manager.cc:902] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:54.849701 9075 tablet_copy_client.cc:308] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:00:54.850176 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet f8b939db0e5b464dbcecbf8e18cccadf from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:33761)
I20240502 14:00:54.850286 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet f8b939db0e5b464dbcecbf8e18cccadf from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:33761: session id = efdbf7bdfe0b41bdadb1add053b09a7e-f8b939db0e5b464dbcecbf8e18cccadf
I20240502 14:00:54.851311 32143 tablet_copy_source_session.cc:205] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.852486 9075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b939db0e5b464dbcecbf8e18cccadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.854656 9075 tablet_copy_client.cc:783] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.854815 9075 tablet_copy_client.cc:647] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.859918 29451 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:00:54.861116 9075 tablet_copy_client.cc:523] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.862447 9075 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:54.866641 8697 raft_consensus.cc:482] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:54.866737 8697 raft_consensus.cc:504] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.867132 8697 leader_election.cc:288] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:54.867344 31970 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:54.867729 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2dc222951548708862c3adf205d0ef" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:54.867846 29732 raft_consensus.cc:2454] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:54.868189 31926 leader_election.cc:302] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:54.868305 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2dc222951548708862c3adf205d0ef" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:54.868443 8697 raft_consensus.cc:2781] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:54.868448 29602 raft_consensus.cc:2454] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:54.868495 8697 raft_consensus.cc:482] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:54.868532 8697 raft_consensus.cc:3037] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.869513 8697 raft_consensus.cc:504] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.869884 8697 leader_election.cc:288] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:54.870146 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2dc222951548708862c3adf205d0ef" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:54.870249 29732 raft_consensus.cc:3037] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.870247 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2dc222951548708862c3adf205d0ef" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:54.870340 29602 raft_consensus.cc:3037] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.870481 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:54.871300 29732 raft_consensus.cc:2454] T 9e2dc222951548708862c3adf205d0ef P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:54.871419 29602 raft_consensus.cc:2454] T 9e2dc222951548708862c3adf205d0ef P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:54.871661 31926 leader_election.cc:302] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:54.871804 8697 raft_consensus.cc:2781] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:54.871887 8697 raft_consensus.cc:686] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:54.872013 8697 consensus_queue.cc:237] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.876173 29320 catalog_manager.cc:5484] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.880208 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:54.881572 6975 raft_consensus.cc:482] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:54.881659 6975 raft_consensus.cc:504] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.881989 6975 leader_election.cc:288] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:54.882344 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708b6a4007e1486ba7269d23962891ac" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:54.882462 29864 raft_consensus.cc:2454] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:54.883028 31926 leader_election.cc:302] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:54.883188 6975 raft_consensus.cc:2781] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:54.883250 6975 raft_consensus.cc:482] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:54.883286 6975 raft_consensus.cc:3037] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.884186 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708b6a4007e1486ba7269d23962891ac" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:54.884318 29729 raft_consensus.cc:2454] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:54.884378 6975 raft_consensus.cc:504] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.884678 6975 leader_election.cc:288] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:54.885334 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708b6a4007e1486ba7269d23962891ac" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:54.885442 29864 raft_consensus.cc:3037] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.886468 29864 raft_consensus.cc:2454] T 708b6a4007e1486ba7269d23962891ac P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:54.890269 31926 leader_election.cc:302] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:00:54.890745 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708b6a4007e1486ba7269d23962891ac" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:54.890787 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:54.890794 6975 raft_consensus.cc:2781] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:54.890897 6975 raft_consensus.cc:686] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:54.890908 31861 raft_consensus.cc:1203] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:00:54.890942 29729 raft_consensus.cc:3037] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.891031 6975 consensus_queue.cc:237] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:54.892004 29729 raft_consensus.cc:2454] T 708b6a4007e1486ba7269d23962891ac P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:54.893640 29326 catalog_manager.cc:5484] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:54.900260 9075 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.900525 9075 tablet_bootstrap.cc:492] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:00:54.900679 9075 ts_tablet_manager.cc:1367] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.004s
I20240502 14:00:54.901109 9075 raft_consensus.cc:348] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:54.901214 9075 raft_consensus.cc:729] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.901329 9075 consensus_queue.cc:260] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:54.902364 9075 ts_tablet_manager.cc:1398] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:54.903095 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-f8b939db0e5b464dbcecbf8e18cccadf received from {username='slave'} at 127.22.55.131:33761
I20240502 14:00:54.903195 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-f8b939db0e5b464dbcecbf8e18cccadf on tablet f8b939db0e5b464dbcecbf8e18cccadf with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:00:54.904052 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:54.908617 9195 ts_tablet_manager.cc:902] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:54.909060 9195 tablet_copy_client.cc:308] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:00:54.909518 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet 1ea3465ad91540cea8906bcc406efd07 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:39449)
I20240502 14:00:54.909615 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet 1ea3465ad91540cea8906bcc406efd07 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:39449: session id = d6e2e2c10def4331bec17bd110c9b423-1ea3465ad91540cea8906bcc406efd07
I20240502 14:00:54.910655 32275 tablet_copy_source_session.cc:205] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:54.911494 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea3465ad91540cea8906bcc406efd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:54.913483 9195 tablet_copy_client.cc:783] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:54.913599 9195 tablet_copy_client.cc:647] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:54.920295 32101 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:00:54.930811 29864 raft_consensus.cc:1203] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:54.935837 9195 tablet_copy_client.cc:523] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:54.937358 9195 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:54.941471 9195 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:54.941725 9195 tablet_bootstrap.cc:492] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:54.941881 9195 ts_tablet_manager.cc:1367] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:00:54.942320 9195 raft_consensus.cc:348] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.942440 9195 raft_consensus.cc:729] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:54.942576 9195 consensus_queue.cc:260] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.942852 9195 ts_tablet_manager.cc:1398] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:54.944136 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-1ea3465ad91540cea8906bcc406efd07 received from {username='slave'} at 127.22.55.130:39449
I20240502 14:00:54.944226 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-1ea3465ad91540cea8906bcc406efd07 on tablet 1ea3465ad91540cea8906bcc406efd07 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:54.947667 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:54.970189 8814 raft_consensus.cc:482] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:54.970317 8814 raft_consensus.cc:504] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.970695 8814 leader_election.cc:288] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:54.971092 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7f219b594aef92b19cb1a8819ba5" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:54.971235 32254 raft_consensus.cc:2454] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:54.971745 32058 leader_election.cc:302] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:54.971896 8814 raft_consensus.cc:2781] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:54.971948 8814 raft_consensus.cc:482] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:54.971987 8814 raft_consensus.cc:3037] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.971992 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7f219b594aef92b19cb1a8819ba5" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:54.972113 29602 raft_consensus.cc:2454] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:54.972884 8814 raft_consensus.cc:504] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.973187 8814 leader_election.cc:288] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:54.973783 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7f219b594aef92b19cb1a8819ba5" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:54.973882 32254 raft_consensus.cc:3037] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.974725 32254 raft_consensus.cc:2454] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:54.976140 32058 leader_election.cc:302] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:54.976301 8892 raft_consensus.cc:2781] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:54.976380 8892 raft_consensus.cc:686] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:54.976505 8892 consensus_queue.cc:237] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:54.977140 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d7f219b594aef92b19cb1a8819ba5" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:54.977258 29602 raft_consensus.cc:3037] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:54.978183 29602 raft_consensus.cc:2454] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:54.980398 29326 catalog_manager.cc:5484] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:54.984608 29470 raft_consensus.cc:1203] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:54.998498 9244 ts_tablet_manager.cc:902] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:54.999024 9244 tablet_copy_client.cc:308] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:00:54.999464 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet be8afc0891fb43fa9ffc863ad2bc90d2 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:42159)
I20240502 14:00:54.999572 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet be8afc0891fb43fa9ffc863ad2bc90d2 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:42159: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-be8afc0891fb43fa9ffc863ad2bc90d2
I20240502 14:00:55.000496 29884 tablet_copy_source_session.cc:205] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:55.001353 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8afc0891fb43fa9ffc863ad2bc90d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.003749 9244 tablet_copy_client.cc:783] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:55.003942 9244 tablet_copy_client.cc:647] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:55.005687 9244 tablet_copy_client.cc:523] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:55.006831 9244 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:55.012341 9251 ts_tablet_manager.cc:902] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:55.012816 9251 tablet_copy_client.cc:308] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:00:55.013450 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet 5ae0eeb814e34729b6284d838d458ddf from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:59435)
I20240502 14:00:55.013558 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet 5ae0eeb814e34729b6284d838d458ddf from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:59435: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-5ae0eeb814e34729b6284d838d458ddf
I20240502 14:00:55.014549 32275 tablet_copy_source_session.cc:205] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:55.015653 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae0eeb814e34729b6284d838d458ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.017782 9251 tablet_copy_client.cc:783] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:55.018216 9251 tablet_copy_client.cc:647] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:55.021809 9251 tablet_copy_client.cc:523] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:55.023152 9251 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:55.029848 9251 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:55.030097 9251 tablet_bootstrap.cc:492] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:00:55.030285 9251 ts_tablet_manager.cc:1367] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:00:55.030666 9251 raft_consensus.cc:348] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.030764 9251 raft_consensus.cc:729] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.030916 9251 consensus_queue.cc:260] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.031307 9251 ts_tablet_manager.cc:1398] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:55.032028 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-5ae0eeb814e34729b6284d838d458ddf received from {username='slave'} at 127.22.55.132:59435
I20240502 14:00:55.032112 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-5ae0eeb814e34729b6284d838d458ddf on tablet 5ae0eeb814e34729b6284d838d458ddf with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:00:55.038026 9244 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:55.038203 8448 raft_consensus.cc:482] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:55.038327 9244 tablet_bootstrap.cc:492] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:55.038295 8448 raft_consensus.cc:504] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.038496 9244 ts_tablet_manager.cc:1367] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.012s
I20240502 14:00:55.038658 8448 leader_election.cc:288] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:55.039011 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266aa0d4f078466a820bd48e60d044aa" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:55.039011 9244 raft_consensus.cc:348] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:55.039047 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266aa0d4f078466a820bd48e60d044aa" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:55.039139 9244 raft_consensus.cc:729] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.039142 32254 raft_consensus.cc:2454] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:55.039181 32122 raft_consensus.cc:2454] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:55.039283 9244 consensus_queue.cc:260] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:55.039417 29404 leader_election.cc:302] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:55.039566 8448 raft_consensus.cc:2781] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:55.039606 8448 raft_consensus.cc:482] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:55.039637 8448 raft_consensus.cc:3037] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.040400 9244 ts_tablet_manager.cc:1398] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:55.040678 8448 raft_consensus.cc:504] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.040992 8448 leader_election.cc:288] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:00:55.041366 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266aa0d4f078466a820bd48e60d044aa" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:55.041471 32122 raft_consensus.cc:3037] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.041792 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-be8afc0891fb43fa9ffc863ad2bc90d2 received from {username='slave'} at 127.22.55.134:42159
I20240502 14:00:55.041873 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-be8afc0891fb43fa9ffc863ad2bc90d2 on tablet be8afc0891fb43fa9ffc863ad2bc90d2 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:55.042215 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266aa0d4f078466a820bd48e60d044aa" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:55.042311 32254 raft_consensus.cc:3037] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.042572 32122 raft_consensus.cc:2454] T 266aa0d4f078466a820bd48e60d044aa P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:55.042917 29404 leader_election.cc:302] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:55.043159 8655 raft_consensus.cc:2781] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:55.043233 8655 raft_consensus.cc:686] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:55.043318 32254 raft_consensus.cc:2454] T 266aa0d4f078466a820bd48e60d044aa P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:55.043359 8655 consensus_queue.cc:237] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.046669 32122 raft_consensus.cc:1203] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.047233 9258 ts_tablet_manager.cc:902] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:55.047721 9258 tablet_copy_client.cc:308] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.129:44823
I20240502 14:00:55.049726 29320 catalog_manager.cc:5484] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:55.051177 29491 tablet_copy_service.cc:140] P 1d00d3a47c52478894cf8c66959ef7bd: Received BeginTabletCopySession request for tablet fad144c461044c299ea905832908666a from peer 8973f2527acb48f5ad36589652de73e1 ({username='slave'} at 127.22.55.136:57511)
I20240502 14:00:55.051271 29491 tablet_copy_service.cc:161] P 1d00d3a47c52478894cf8c66959ef7bd: Beginning new tablet copy session on tablet fad144c461044c299ea905832908666a from peer 8973f2527acb48f5ad36589652de73e1 at {username='slave'} at 127.22.55.136:57511: session id = 8973f2527acb48f5ad36589652de73e1-fad144c461044c299ea905832908666a
I20240502 14:00:55.052199 29491 tablet_copy_source_session.cc:205] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:55.052973 9258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad144c461044c299ea905832908666a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.055011 9258 tablet_copy_client.cc:783] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:55.055148 9258 tablet_copy_client.cc:647] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:55.056048 9258 tablet_copy_client.cc:523] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:55.056988 9258 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:55.063740 9258 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:55.064011 9258 tablet_bootstrap.cc:492] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: Bootstrap complete.
I20240502 14:00:55.064169 9258 ts_tablet_manager.cc:1367] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:00:55.064640 9258 raft_consensus.cc:348] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:55.064764 9258 raft_consensus.cc:729] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.064915 9258 consensus_queue.cc:260] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:55.065274 9258 ts_tablet_manager.cc:1398] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:55.065765 29491 tablet_copy_service.cc:342] P 1d00d3a47c52478894cf8c66959ef7bd: Request end of tablet copy session 8973f2527acb48f5ad36589652de73e1-fad144c461044c299ea905832908666a received from {username='slave'} at 127.22.55.136:57511
I20240502 14:00:55.065840 29491 tablet_copy_service.cc:433] P 1d00d3a47c52478894cf8c66959ef7bd: ending tablet copy session 8973f2527acb48f5ad36589652de73e1-fad144c461044c299ea905832908666a on tablet fad144c461044c299ea905832908666a with peer 8973f2527acb48f5ad36589652de73e1
I20240502 14:00:55.073122 29602 raft_consensus.cc:1203] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.086320 32119 raft_consensus.cc:1203] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:00:55.092905 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:55.100370 29602 raft_consensus.cc:1203] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.101181 29602 raft_consensus.cc:1261] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:55.101572 9295 consensus_queue.cc:1035] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.102043 32254 raft_consensus.cc:1261] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:55.103614 8814 consensus_queue.cc:1035] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.104096 9294 mvcc.cc:205] Tried to move back new op lower bound from 7023241032091222016 to 7023241031585316864. Current Snapshot: MvccSnapshot[applied={T|T < 7023241032091222016}]
I20240502 14:00:55.105767 9297 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:55.106734 9297 tablet.cc:1620] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:55.107342 29326 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 370d7f219b594aef92b19cb1a8819ba5 (table rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295]). Expected version 1 got 0
I20240502 14:00:55.111155 9300 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:55.111460 9300 tablet.cc:1620] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:55.112321 29326 catalog_manager.cc:5576] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041) does not have the latest schema for tablet 370d7f219b594aef92b19cb1a8819ba5 (table rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295]). Expected version 1 got 0
I20240502 14:00:55.113118 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295] alter complete (version 1)
I20240502 14:00:55.113864 29328 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet 370d7f219b594aef92b19cb1a8819ba5 (table rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295]). Expected version 1 got 0
I20240502 14:00:55.115581 9298 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:55.115963 9298 tablet.cc:1620] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:55.130228 29470 raft_consensus.cc:1203] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.148200 9195 ts_tablet_manager.cc:902] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:55.148742 9195 tablet_copy_client.cc:308] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:00:55.149411 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet 768df43846d94951a3a48f6f626bb473 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:37987)
I20240502 14:00:55.149529 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet 768df43846d94951a3a48f6f626bb473 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:37987: session id = d6e2e2c10def4331bec17bd110c9b423-768df43846d94951a3a48f6f626bb473
I20240502 14:00:55.150558 32012 tablet_copy_source_session.cc:205] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:55.151471 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768df43846d94951a3a48f6f626bb473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.153829 9195 tablet_copy_client.cc:783] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:55.153990 9195 tablet_copy_client.cc:647] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:55.156888 9195 tablet_copy_client.cc:523] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:55.158494 9195 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:55.165218 9244 ts_tablet_manager.cc:902] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:55.165660 9244 tablet_copy_client.cc:308] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:00:55.166087 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:44335)
I20240502 14:00:55.166213 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet 3afd58ca519e4bc8a6ab00c37abf9f1a from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:44335: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-3afd58ca519e4bc8a6ab00c37abf9f1a
I20240502 14:00:55.167333 31865 tablet_copy_source_session.cc:205] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:55.168329 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.171087 9244 tablet_copy_client.cc:783] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:55.171247 9244 tablet_copy_client.cc:647] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:55.171489 29470 raft_consensus.cc:1203] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.173249 9244 tablet_copy_client.cc:523] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:55.175231 9244 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:55.176115 29470 raft_consensus.cc:1203] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.176513 9195 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:55.176800 9195 tablet_bootstrap.cc:492] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:00:55.176976 9195 ts_tablet_manager.cc:1367] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240502 14:00:55.177459 9195 raft_consensus.cc:348] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:55.177603 9195 raft_consensus.cc:729] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.177747 9195 consensus_queue.cc:260] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:55.178068 9195 ts_tablet_manager.cc:1398] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 14:00:55.178704 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-768df43846d94951a3a48f6f626bb473 received from {username='slave'} at 127.22.55.130:37987
I20240502 14:00:55.178762 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-768df43846d94951a3a48f6f626bb473 on tablet 768df43846d94951a3a48f6f626bb473 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:00:55.185909 9244 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:55.186163 9244 tablet_bootstrap.cc:492] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:00:55.186303 9244 ts_tablet_manager.cc:1367] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 14:00:55.186712 9244 raft_consensus.cc:348] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:55.186817 9244 raft_consensus.cc:729] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.187178 9244 consensus_queue.cc:260] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:55.187533 9244 ts_tablet_manager.cc:1398] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:55.188208 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-3afd58ca519e4bc8a6ab00c37abf9f1a received from {username='slave'} at 127.22.55.134:44335
I20240502 14:00:55.188293 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-3afd58ca519e4bc8a6ab00c37abf9f1a on tablet 3afd58ca519e4bc8a6ab00c37abf9f1a with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:00:55.205102 29729 raft_consensus.cc:1203] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.220470 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:55.228586 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:55.230934 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:55.232666 9297 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:55.232996 9297 tablet.cc:1620] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:55.234400 9300 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:55.234711 9300 tablet.cc:1620] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:55.235287 9298 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:55.235589 9298 tablet.cc:1620] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: MemRowSet was empty: no flush needed.
I20240502 14:00:55.242274 29451 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:55.242465 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=0684c95e4c5d4954adf00cc03f5b5295] alter complete (version 2)
I20240502 14:00:55.242540 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:55.252077 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:55.260552 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:55.266314 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:55.281898 32101 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:00:55.332262 8404 consensus_queue.cc:1035] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.339408 8697 consensus_queue.cc:1035] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.339867 29600 raft_consensus.cc:1203] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.347069 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:55.352250 9297 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:55.353080 9298 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:55.354960 9300 tablet.cc:1722] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:55.360862 29325 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=0684c95e4c5d4954adf00cc03f5b5295] alter complete (version 3)
I20240502 14:00:55.361629 29600 raft_consensus.cc:1203] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.420836 29732 raft_consensus.cc:1203] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.429383 31992 raft_consensus.cc:1203] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:00:55.441990 8448 consensus_queue.cc:1035] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.447959 8655 consensus_queue.cc:1035] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.464520 29321 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:55.464715 29321 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:55.468447 29321 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet 370d7f219b594aef92b19cb1a8819ba5
I20240502 14:00:55.468961 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 370d7f219b594aef92b19cb1a8819ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:55 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:55.469226 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 370d7f219b594aef92b19cb1a8819ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:55 UTC) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:55.469282 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 370d7f219b594aef92b19cb1a8819ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:55 UTC) from {username='slave'} at 127.0.0.1:55288
I20240502 14:00:55.469684 9374 tablet_replica.cc:330] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:00:55.469918 9372 tablet_replica.cc:330] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:55.469915 9374 raft_consensus.cc:2229] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:55.470044 9373 tablet_replica.cc:330] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:55.470054 9374 raft_consensus.cc:2258] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:55.470527 8697 consensus_queue.cc:1035] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.471009 9372 raft_consensus.cc:2229] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:55.471179 9372 raft_consensus.cc:2258] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:55.471403 9374 ts_tablet_manager.cc:1875] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:55.471449 9373 raft_consensus.cc:2229] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:55.471686 9373 raft_consensus.cc:2258] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:55.472332 9373 ts_tablet_manager.cc:1875] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:55.473599 9372 ts_tablet_manager.cc:1875] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:55.475904 8697 consensus_queue.cc:1035] T 708b6a4007e1486ba7269d23962891ac P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.480762 9372 ts_tablet_manager.cc:1888] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:55.483129 9373 ts_tablet_manager.cc:1888] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:55.483294 9373 log.cc:1196] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/370d7f219b594aef92b19cb1a8819ba5
I20240502 14:00:55.483834 9374 ts_tablet_manager.cc:1888] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:55.483919 9374 log.cc:1196] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/370d7f219b594aef92b19cb1a8819ba5
I20240502 14:00:55.484022 9372 log.cc:1196] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/370d7f219b594aef92b19cb1a8819ba5
I20240502 14:00:55.484632 9372 ts_tablet_manager.cc:1909] T 370d7f219b594aef92b19cb1a8819ba5 P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:55.484632 9373 ts_tablet_manager.cc:1909] T 370d7f219b594aef92b19cb1a8819ba5 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting consensus metadata
I20240502 14:00:55.484992 9374 ts_tablet_manager.cc:1909] T 370d7f219b594aef92b19cb1a8819ba5 P d6e2e2c10def4331bec17bd110c9b423: Deleting consensus metadata
I20240502 14:00:55.487229 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 370d7f219b594aef92b19cb1a8819ba5 (table rebalancer_dynamic_table_new_name [id=0684c95e4c5d4954adf00cc03f5b5295]) successfully deleted
I20240502 14:00:55.487272 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 370d7f219b594aef92b19cb1a8819ba5 (table rebalancer_dynamic_table_new_name [id=0684c95e4c5d4954adf00cc03f5b5295]) successfully deleted
I20240502 14:00:55.487470 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 370d7f219b594aef92b19cb1a8819ba5 (table rebalancer_dynamic_table_new_name [id=0684c95e4c5d4954adf00cc03f5b5295]) successfully deleted
I20240502 14:00:55.506732 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:55.507090 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:55.510288 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:55.511324 29451 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:55.511565 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:55.523378 32101 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:00:55.527765 29863 raft_consensus.cc:1203] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.535583 29713 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:55.559752 31992 raft_consensus.cc:1203] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.562753 31840 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:55.571715 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:55.579475 29451 tablet_service.cc:1467] Processing CreateTablet for tablet f2478cbf464a453a93bcd6edefdb5c1e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1965b3e77e824b95be95c7699523c627]), partition=
I20240502 14:00:55.579475 32233 tablet_service.cc:1467] Processing CreateTablet for tablet f2478cbf464a453a93bcd6edefdb5c1e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1965b3e77e824b95be95c7699523c627]), partition=
I20240502 14:00:55.579758 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2478cbf464a453a93bcd6edefdb5c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.579758 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2478cbf464a453a93bcd6edefdb5c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.582262 9389 tablet_bootstrap.cc:492] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:55.582402 31970 tablet_service.cc:1467] Processing CreateTablet for tablet f2478cbf464a453a93bcd6edefdb5c1e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1965b3e77e824b95be95c7699523c627]), partition=
I20240502 14:00:55.582648 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2478cbf464a453a93bcd6edefdb5c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.583346 9389 tablet_bootstrap.cc:654] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.583746 9390 tablet_bootstrap.cc:492] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:55.584877 9390 tablet_bootstrap.cc:654] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.590770 9394 tablet_bootstrap.cc:492] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:55.591866 9394 tablet_bootstrap.cc:654] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.592890 9394 tablet_bootstrap.cc:492] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:55.592979 9394 ts_tablet_manager.cc:1367] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:55.593482 9394 raft_consensus.cc:348] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.593689 9394 raft_consensus.cc:374] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.593778 9394 raft_consensus.cc:729] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.593909 9394 consensus_queue.cc:260] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.594503 29326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:55.595343 9394 ts_tablet_manager.cc:1398] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:55.595543 9389 tablet_bootstrap.cc:492] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:55.595644 9389 ts_tablet_manager.cc:1367] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:00:55.595969 9390 tablet_bootstrap.cc:492] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:55.596086 9389 raft_consensus.cc:348] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.596184 9389 raft_consensus.cc:374] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.596215 9389 raft_consensus.cc:729] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.596347 9390 ts_tablet_manager.cc:1367] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:00:55.596330 9389 consensus_queue.cc:260] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.596611 9389 ts_tablet_manager.cc:1398] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:55.596825 9390 raft_consensus.cc:348] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.596928 9390 raft_consensus.cc:374] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.596963 9390 raft_consensus.cc:729] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.599190 9390 consensus_queue.cc:260] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.599637 9390 ts_tablet_manager.cc:1398] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:55.601032 29600 raft_consensus.cc:1203] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.608669 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 282d8021fee14b309b4a357f370aa334 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.608758 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 2868541b54c447219db948c92b9a132b (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.608994 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282d8021fee14b309b4a357f370aa334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.609056 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2868541b54c447219db948c92b9a132b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.609285 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 282d8021fee14b309b4a357f370aa334 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.609563 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282d8021fee14b309b4a357f370aa334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.611541 9389 tablet_bootstrap.cc:492] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:55.612231 29713 tablet_service.cc:1467] Processing CreateTablet for tablet 282d8021fee14b309b4a357f370aa334 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.612571 9389 tablet_bootstrap.cc:654] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.612542 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 1fefaee1ab4c4a77aa7ed79d6c2e8894 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.612594 29713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 282d8021fee14b309b4a357f370aa334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.612779 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fefaee1ab4c4a77aa7ed79d6c2e8894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.613479 31840 tablet_service.cc:1467] Processing CreateTablet for tablet 1fefaee1ab4c4a77aa7ed79d6c2e8894 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.613708 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fefaee1ab4c4a77aa7ed79d6c2e8894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.614967 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 2868541b54c447219db948c92b9a132b (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.615059 29582 tablet_service.cc:1467] Processing CreateTablet for tablet 2868541b54c447219db948c92b9a132b (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.615237 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2868541b54c447219db948c92b9a132b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.615296 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2868541b54c447219db948c92b9a132b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.616024 9400 tablet_bootstrap.cc:492] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:55.616915 9400 tablet_bootstrap.cc:654] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.617950 29843 tablet_service.cc:1467] Processing CreateTablet for tablet 1fefaee1ab4c4a77aa7ed79d6c2e8894 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=833e2c3452fd469a82e72f7cd56bc611]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:55.618211 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fefaee1ab4c4a77aa7ed79d6c2e8894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:55.618999 9389 tablet_bootstrap.cc:492] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:55.619093 9389 ts_tablet_manager.cc:1367] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:55.619544 9389 raft_consensus.cc:348] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.619649 9389 raft_consensus.cc:374] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.619680 9389 raft_consensus.cc:729] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.619787 9389 consensus_queue.cc:260] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.620061 9389 ts_tablet_manager.cc:1398] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:55.620584 9404 tablet_bootstrap.cc:492] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:55.621178 9400 tablet_bootstrap.cc:492] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:55.621340 9400 ts_tablet_manager.cc:1367] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:55.621621 9405 tablet_bootstrap.cc:492] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:55.621874 9400 raft_consensus.cc:348] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.621995 9400 raft_consensus.cc:374] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.622071 9400 raft_consensus.cc:729] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.622231 9400 consensus_queue.cc:260] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.622457 9405 tablet_bootstrap.cc:654] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.622561 9400 ts_tablet_manager.cc:1398] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:55.622707 9400 tablet_bootstrap.cc:492] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:55.623275 9404 tablet_bootstrap.cc:654] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.623651 9405 tablet_bootstrap.cc:492] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:55.623725 9400 tablet_bootstrap.cc:654] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.623839 9405 ts_tablet_manager.cc:1367] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:55.624593 9403 tablet_bootstrap.cc:492] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:55.625543 9403 tablet_bootstrap.cc:654] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.625756 9405 raft_consensus.cc:348] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.625872 9400 tablet_bootstrap.cc:492] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:55.625916 9405 raft_consensus.cc:374] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.625991 9400 ts_tablet_manager.cc:1367] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:55.626013 9405 raft_consensus.cc:729] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.626222 9401 tablet_bootstrap.cc:492] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:55.626485 9400 raft_consensus.cc:348] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.626636 9400 raft_consensus.cc:374] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.626714 9400 raft_consensus.cc:729] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.626336 9405 consensus_queue.cc:260] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.627391 9405 ts_tablet_manager.cc:1398] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:55.627566 9401 tablet_bootstrap.cc:654] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.627606 9400 consensus_queue.cc:260] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.628016 9400 ts_tablet_manager.cc:1398] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:55.628842 9403 tablet_bootstrap.cc:492] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:55.629057 9404 tablet_bootstrap.cc:492] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:55.629141 9404 ts_tablet_manager.cc:1367] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:55.629484 9401 tablet_bootstrap.cc:492] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:55.629552 9401 ts_tablet_manager.cc:1367] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:55.628952 9403 ts_tablet_manager.cc:1367] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:55.629794 9404 raft_consensus.cc:348] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.629947 9404 raft_consensus.cc:374] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.629998 9404 raft_consensus.cc:729] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.630115 9404 consensus_queue.cc:260] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.630405 9404 ts_tablet_manager.cc:1398] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:55.630597 9403 raft_consensus.cc:348] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.630755 9403 raft_consensus.cc:374] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.630860 9403 raft_consensus.cc:729] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.631089 9403 consensus_queue.cc:260] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.631567 9403 ts_tablet_manager.cc:1398] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:55.631749 9403 tablet_bootstrap.cc:492] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:00:55.632627 9403 tablet_bootstrap.cc:654] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.633757 9403 tablet_bootstrap.cc:492] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e: No bootstrap required, opened a new log
I20240502 14:00:55.633885 9403 ts_tablet_manager.cc:1367] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:55.634346 9403 raft_consensus.cc:348] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.634613 9403 raft_consensus.cc:374] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.634704 9403 raft_consensus.cc:729] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.634770 9401 raft_consensus.cc:348] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.634912 9401 raft_consensus.cc:374] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.634958 9401 raft_consensus.cc:729] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.635036 9403 consensus_queue.cc:260] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.635079 9401 consensus_queue.cc:260] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.635429 9401 ts_tablet_manager.cc:1398] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:55.635435 9403 ts_tablet_manager.cc:1398] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:55.635577 9401 tablet_bootstrap.cc:492] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:55.636463 9401 tablet_bootstrap.cc:654] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:55.637967 9401 tablet_bootstrap.cc:492] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:55.638103 9401 ts_tablet_manager.cc:1367] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:55.639593 9401 raft_consensus.cc:348] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.639755 9401 raft_consensus.cc:374] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:55.639832 9401 raft_consensus.cc:729] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:55.640029 9401 consensus_queue.cc:260] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.640386 9401 ts_tablet_manager.cc:1398] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:55.641304 29863 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2"
dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48564
I20240502 14:00:55.641433 29863 raft_consensus.cc:595] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:55.646431 32255 raft_consensus.cc:1203] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:55.668882 8404 raft_consensus.cc:482] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:55.668990 8404 raft_consensus.cc:504] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.669411 8404 leader_election.cc:288] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:55.669809 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2478cbf464a453a93bcd6edefdb5c1e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:55.669991 32255 raft_consensus.cc:2454] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:55.669986 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2478cbf464a453a93bcd6edefdb5c1e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:00:55.670128 29470 raft_consensus.cc:2454] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 0.
I20240502 14:00:55.670734 31927 leader_election.cc:302] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:55.670922 8404 raft_consensus.cc:2781] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:55.671015 8404 raft_consensus.cc:482] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:55.671072 8404 raft_consensus.cc:3037] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.672240 8404 raft_consensus.cc:504] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.672549 8404 leader_election.cc:288] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:00:55.672860 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2478cbf464a453a93bcd6edefdb5c1e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:00:55.672959 29470 raft_consensus.cc:3037] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.673271 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2478cbf464a453a93bcd6edefdb5c1e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:55.673373 32254 raft_consensus.cc:3037] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.673980 29470 raft_consensus.cc:2454] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:55.674407 32254 raft_consensus.cc:2454] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:00:55.674997 31927 leader_election.cc:302] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:55.675597 8697 raft_consensus.cc:2781] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:55.675694 8697 raft_consensus.cc:686] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:00:55.675822 8697 consensus_queue.cc:237] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:55.678462 29320 catalog_manager.cc:5484] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:55.772400 6614 raft_consensus.cc:482] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:55.772543 6614 raft_consensus.cc:504] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.772996 6614 leader_election.cc:288] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:55.774366 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fefaee1ab4c4a77aa7ed79d6c2e8894" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:00:55.774497 29732 raft_consensus.cc:2454] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:55.775188 31795 leader_election.cc:302] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:55.775352 6614 raft_consensus.cc:2781] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:55.775436 6614 raft_consensus.cc:482] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:55.775477 6614 raft_consensus.cc:3037] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.775888 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fefaee1ab4c4a77aa7ed79d6c2e8894" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:55.776006 29863 raft_consensus.cc:2454] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 0.
I20240502 14:00:55.776624 6614 raft_consensus.cc:504] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.776885 6614 leader_election.cc:288] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:00:55.777321 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fefaee1ab4c4a77aa7ed79d6c2e8894" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:55.777282 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fefaee1ab4c4a77aa7ed79d6c2e8894" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:55.777419 29863 raft_consensus.cc:3037] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.777419 29732 raft_consensus.cc:3037] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.778353 29863 raft_consensus.cc:2454] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:55.778353 29732 raft_consensus.cc:2454] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:00:55.778707 31795 leader_election.cc:302] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:55.779099 6614 raft_consensus.cc:2781] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:55.779186 6614 raft_consensus.cc:686] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:00:55.779322 6614 consensus_queue.cc:237] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:00:55.782321 29320 catalog_manager.cc:5484] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:00:55.791002 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:55.794308 29470 raft_consensus.cc:1261] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:55.794319 32255 raft_consensus.cc:1261] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:55.794781 8697 consensus_queue.cc:1035] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.795081 8404 consensus_queue.cc:1035] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:55.797817 29320 catalog_manager.cc:5576] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823) does not have the latest schema for tablet f2478cbf464a453a93bcd6edefdb5c1e (table rebalancer_dynamic_table [id=1965b3e77e824b95be95c7699523c627]). Expected version 1 got 0
I20240502 14:00:55.798456 9456 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:55.798818 9456 tablet.cc:1620] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:55.799641 29323 catalog_manager.cc:5576] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201) does not have the latest schema for tablet f2478cbf464a453a93bcd6edefdb5c1e (table rebalancer_dynamic_table [id=1965b3e77e824b95be95c7699523c627]). Expected version 1 got 0
I20240502 14:00:55.799665 9454 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:55.801999 9300 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:55.802268 9300 tablet.cc:1620] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:55.802590 9454 tablet.cc:1620] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:55.803684 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=1965b3e77e824b95be95c7699523c627] alter complete (version 1)
I20240502 14:00:55.821810 8736 raft_consensus.cc:979] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: : Instructing follower efdbf7bdfe0b41bdadb1add053b09a7e to start an election
I20240502 14:00:55.821911 8736 raft_consensus.cc:1067] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Signalling peer efdbf7bdfe0b41bdadb1add053b09a7e to start an election
I20240502 14:00:55.822454 29732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2"
dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
from {username='slave'} at 127.22.55.132:54233
I20240502 14:00:55.822619 29732 raft_consensus.cc:482] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:55.822676 29732 raft_consensus.cc:3037] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:55.823479 29732 raft_consensus.cc:504] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:55.823844 29732 leader_election.cc:288] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:55.824038 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:55.824141 29863 raft_consensus.cc:3032] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:55.824182 29863 raft_consensus.cc:729] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:55.824219 32119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:55.824330 32119 raft_consensus.cc:3037] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:55.824339 29863 consensus_queue.cc:260] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:55.824527 29863 raft_consensus.cc:3037] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:55.825289 32119 raft_consensus.cc:2454] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
I20240502 14:00:55.825455 29863 raft_consensus.cc:2454] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
I20240502 14:00:55.825768 29667 leader_election.cc:302] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:55.825785 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8afc0891fb43fa9ffc863ad2bc90d2" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:55.825891 31992 raft_consensus.cc:3037] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:55.826018 8756 raft_consensus.cc:2781] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:55.826099 8756 raft_consensus.cc:686] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:55.826223 8756 consensus_queue.cc:237] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:55.826797 31992 raft_consensus.cc:2454] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
I20240502 14:00:55.828539 29329 catalog_manager.cc:5484] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 1 to 2, leader changed from 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 2 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:55.871387 9295 raft_consensus.cc:482] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:55.871485 9295 raft_consensus.cc:504] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.871836 9295 leader_election.cc:288] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:55.872105 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2868541b54c447219db948c92b9a132b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:55.872264 32254 raft_consensus.cc:2454] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:55.872270 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2868541b54c447219db948c92b9a132b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:55.872431 29600 raft_consensus.cc:2454] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:55.872524 32058 leader_election.cc:302] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:55.872696 9295 raft_consensus.cc:2781] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:55.872742 9295 raft_consensus.cc:482] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:55.872779 9295 raft_consensus.cc:3037] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.873855 9295 raft_consensus.cc:504] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.874145 9295 leader_election.cc:288] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:00:55.874449 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2868541b54c447219db948c92b9a132b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:55.874573 32254 raft_consensus.cc:3037] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.874972 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2868541b54c447219db948c92b9a132b" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:55.875077 29600 raft_consensus.cc:3037] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.875478 32254 raft_consensus.cc:2454] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:55.875742 32058 leader_election.cc:302] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:55.875906 9295 raft_consensus.cc:2781] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:55.875947 29600 raft_consensus.cc:2454] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:55.876083 9295 raft_consensus.cc:686] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:55.876214 9295 consensus_queue.cc:237] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:55.878497 29323 catalog_manager.cc:5484] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:55.907426 29323 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:55.912092 9454 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:55.912434 9454 tablet.cc:1620] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: MemRowSet was empty: no flush needed.
I20240502 14:00:55.912870 9456 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:55.913174 9456 tablet.cc:1620] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: MemRowSet was empty: no flush needed.
I20240502 14:00:55.914687 9300 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:55.914988 9300 tablet.cc:1620] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: MemRowSet was empty: no flush needed.
I20240502 14:00:55.919160 29323 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=1965b3e77e824b95be95c7699523c627] alter complete (version 2)
I20240502 14:00:55.946339 8233 raft_consensus.cc:482] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:55.946442 8233 raft_consensus.cc:504] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.946806 8233 leader_election.cc:288] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:55.947032 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d8021fee14b309b4a357f370aa334" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:00:55.947173 29863 raft_consensus.cc:2454] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:55.947373 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d8021fee14b309b4a357f370aa334" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:55.947444 29667 leader_election.cc:302] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:55.947495 31859 raft_consensus.cc:2454] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 0.
I20240502 14:00:55.947588 8233 raft_consensus.cc:2781] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:55.947626 8233 raft_consensus.cc:482] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:55.947667 8233 raft_consensus.cc:3037] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.948664 8233 raft_consensus.cc:504] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.948976 8233 leader_election.cc:288] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:55.949239 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d8021fee14b309b4a357f370aa334" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:55.949352 31861 raft_consensus.cc:3037] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.949363 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d8021fee14b309b4a357f370aa334" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:00:55.949457 29863 raft_consensus.cc:3037] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:55.950369 29863 raft_consensus.cc:2454] T 282d8021fee14b309b4a357f370aa334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:55.950377 31861 raft_consensus.cc:2454] T 282d8021fee14b309b4a357f370aa334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:00:55.950650 29667 leader_election.cc:302] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:00:55.950886 8233 raft_consensus.cc:2781] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:55.951013 8233 raft_consensus.cc:686] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:55.951249 8233 consensus_queue.cc:237] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:55.953560 29320 catalog_manager.cc:5484] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:55.970436 29843 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:55.975265 31970 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:55.979133 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:55.982996 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:55.983407 29451 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:55.992765 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:55.996359 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:56.014425 32101 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:00:56.022392 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:56.035813 9454 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:56.036783 9300 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:56.039170 9456 tablet.cc:1722] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:56.054167 29320 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=1965b3e77e824b95be95c7699523c627] alter complete (version 3)
I20240502 14:00:56.156929 29328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:56.157073 29328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:00:56.159322 29328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: Sending DeleteTablet for 3 replicas of tablet f2478cbf464a453a93bcd6edefdb5c1e
I20240502 14:00:56.159869 31970 tablet_service.cc:1514] Processing DeleteTablet for tablet f2478cbf464a453a93bcd6edefdb5c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:56 UTC) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:56.160185 9513 tablet_replica.cc:330] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:56.160578 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet f2478cbf464a453a93bcd6edefdb5c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:56 UTC) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:56.160648 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet f2478cbf464a453a93bcd6edefdb5c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:00:56 UTC) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:56.160916 9515 tablet_replica.cc:330] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:56.160948 9513 raft_consensus.cc:2229] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:00:56.161154 9513 raft_consensus.cc:2258] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:56.166492 9513 ts_tablet_manager.cc:1875] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:56.166551 9514 tablet_replica.cc:330] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:56.166738 9514 raft_consensus.cc:2229] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:56.166931 9514 raft_consensus.cc:2258] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:56.167400 9515 raft_consensus.cc:2229] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:56.168025 9514 ts_tablet_manager.cc:1875] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:56.168296 9515 raft_consensus.cc:2258] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:56.171195 9513 ts_tablet_manager.cc:1888] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:56.171311 9513 log.cc:1196] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/f2478cbf464a453a93bcd6edefdb5c1e
I20240502 14:00:56.171631 9513 ts_tablet_manager.cc:1909] T f2478cbf464a453a93bcd6edefdb5c1e P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting consensus metadata
I20240502 14:00:56.173079 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet f2478cbf464a453a93bcd6edefdb5c1e (table rebalancer_dynamic_table_new_name [id=1965b3e77e824b95be95c7699523c627]) successfully deleted
I20240502 14:00:56.174124 9514 ts_tablet_manager.cc:1888] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:56.174252 9514 log.cc:1196] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/f2478cbf464a453a93bcd6edefdb5c1e
I20240502 14:00:56.174588 9514 ts_tablet_manager.cc:1909] T f2478cbf464a453a93bcd6edefdb5c1e P 8973f2527acb48f5ad36589652de73e1: Deleting consensus metadata
I20240502 14:00:56.174726 9515 ts_tablet_manager.cc:1875] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:00:56.175220 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet f2478cbf464a453a93bcd6edefdb5c1e (table rebalancer_dynamic_table_new_name [id=1965b3e77e824b95be95c7699523c627]) successfully deleted
I20240502 14:00:56.180322 9515 ts_tablet_manager.cc:1888] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:00:56.181113 9515 log.cc:1196] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2478cbf464a453a93bcd6edefdb5c1e
I20240502 14:00:56.182411 9515 ts_tablet_manager.cc:1909] T f2478cbf464a453a93bcd6edefdb5c1e P 1d00d3a47c52478894cf8c66959ef7bd: Deleting consensus metadata
I20240502 14:00:56.187254 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet f2478cbf464a453a93bcd6edefdb5c1e (table rebalancer_dynamic_table_new_name [id=1965b3e77e824b95be95c7699523c627]) successfully deleted
I20240502 14:00:56.208447 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:56.211356 32120 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:00:56.211779 8756 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:56.214392 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:56.215889 29843 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:00:56.229684 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:56.232772 29451 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:56.236927 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:56.244503 6614 consensus_queue.cc:1035] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.247465 32101 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 14:00:56.249747 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:56.261219 29324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:00:56.270292 29843 tablet_service.cc:1467] Processing CreateTablet for tablet a4a635382b94471088de77b6c431d334 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2]), partition=
I20240502 14:00:56.270587 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a635382b94471088de77b6c431d334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.271842 31840 tablet_service.cc:1467] Processing CreateTablet for tablet a4a635382b94471088de77b6c431d334 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2]), partition=
I20240502 14:00:56.272092 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a635382b94471088de77b6c431d334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.273075 32101 tablet_service.cc:1467] Processing CreateTablet for tablet a4a635382b94471088de77b6c431d334 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2]), partition=
I20240502 14:00:56.273325 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a635382b94471088de77b6c431d334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.274152 29864 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:00:56.275841 8233 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:56.279361 9531 tablet_bootstrap.cc:492] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:56.280272 8814 consensus_queue.cc:1035] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.279491 6614 consensus_queue.cc:1035] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.286042 9530 tablet_bootstrap.cc:492] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:00:56.287905 31990 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:00:56.288890 8756 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:56.291368 9531 tablet_bootstrap.cc:654] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.295032 9530 tablet_bootstrap.cc:654] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.295030 9295 consensus_queue.cc:1035] T 2868541b54c447219db948c92b9a132b P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.300431 9530 tablet_bootstrap.cc:492] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: No bootstrap required, opened a new log
I20240502 14:00:56.300526 9530 ts_tablet_manager.cc:1367] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240502 14:00:56.300930 9530 raft_consensus.cc:348] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.301026 9530 raft_consensus.cc:374] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.301084 9530 raft_consensus.cc:729] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.301225 9530 consensus_queue.cc:260] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.301704 9530 ts_tablet_manager.cc:1398] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:56.301795 9531 tablet_bootstrap.cc:492] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:56.301889 9531 ts_tablet_manager.cc:1367] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:00:56.302386 9531 raft_consensus.cc:348] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.302533 9531 raft_consensus.cc:374] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.302579 9531 raft_consensus.cc:729] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.302696 9531 consensus_queue.cc:260] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.303030 9531 ts_tablet_manager.cc:1398] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:56.317137 9532 tablet_bootstrap.cc:492] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:56.318010 9532 tablet_bootstrap.cc:654] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.319242 9532 tablet_bootstrap.cc:492] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:56.319496 9532 ts_tablet_manager.cc:1367] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:56.319891 9532 raft_consensus.cc:348] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.319980 9532 raft_consensus.cc:374] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.320015 9532 raft_consensus.cc:729] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.320173 9532 consensus_queue.cc:260] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.321349 9532 ts_tablet_manager.cc:1398] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:56.365864 29470 consensus_queue.cc:237] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.367133 32253 raft_consensus.cc:1261] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:56.367513 31861 raft_consensus.cc:1261] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:56.367561 8448 consensus_queue.cc:1035] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:56.368913 8448 consensus_queue.cc:1035] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:56.369562 8655 raft_consensus.cc:2932] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:56.369951 32253 raft_consensus.cc:2932] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:56.372030 31861 raft_consensus.cc:2932] T fad144c461044c299ea905832908666a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:56.374538 29322 catalog_manager.cc:5484] T fad144c461044c299ea905832908666a P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:56.377431 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet fad144c461044c299ea905832908666a with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54432
I20240502 14:00:56.377779 9570 tablet_replica.cc:330] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:00:56.378923 9570 raft_consensus.cc:2229] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:56.379139 9570 raft_consensus.cc:2258] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:56.379668 9570 ts_tablet_manager.cc:1875] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:56.381945 9570 ts_tablet_manager.cc:1888] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:56.382022 9570 log.cc:1196] T fad144c461044c299ea905832908666a P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/fad144c461044c299ea905832908666a
I20240502 14:00:56.382627 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet fad144c461044c299ea905832908666a (table rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]) successfully deleted
I20240502 14:00:56.429338 9471 raft_consensus.cc:482] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:56.429476 9471 raft_consensus.cc:504] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.429914 9471 leader_election.cc:288] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:56.430500 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a635382b94471088de77b6c431d334" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" is_pre_election: true
I20240502 14:00:56.430622 32120 raft_consensus.cc:2454] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:56.430606 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a635382b94471088de77b6c431d334" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:56.430755 31861 raft_consensus.cc:2454] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 0.
I20240502 14:00:56.431151 29797 leader_election.cc:302] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:56.431380 8739 raft_consensus.cc:2781] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:56.431439 8739 raft_consensus.cc:482] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:56.431481 8739 raft_consensus.cc:3037] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.432682 8739 raft_consensus.cc:504] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.433027 8739 leader_election.cc:288] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:56.433279 32122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a635382b94471088de77b6c431d334" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb"
I20240502 14:00:56.433337 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a635382b94471088de77b6c431d334" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:56.433398 32122 raft_consensus.cc:3037] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.433457 31861 raft_consensus.cc:3037] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.434446 31861 raft_consensus.cc:2454] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:56.434494 32122 raft_consensus.cc:2454] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:00:56.434824 29797 leader_election.cc:302] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:56.435340 8739 raft_consensus.cc:2781] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:56.435423 8739 raft_consensus.cc:686] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:56.435544 8739 consensus_queue.cc:237] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.439164 29320 catalog_manager.cc:5484] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:56.479610 8756 consensus_queue.cc:1035] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.483382 9539 consensus_queue.cc:1035] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.489656 29325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40958:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:56.501556 32101 tablet_service.cc:1467] Processing CreateTablet for tablet 22543bedc3f043bba1d05b675a8f3c75 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.501681 31970 tablet_service.cc:1467] Processing CreateTablet for tablet 22543bedc3f043bba1d05b675a8f3c75 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.501816 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22543bedc3f043bba1d05b675a8f3c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.501945 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22543bedc3f043bba1d05b675a8f3c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.502483 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 22543bedc3f043bba1d05b675a8f3c75 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.502713 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22543bedc3f043bba1d05b675a8f3c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.504012 9532 tablet_bootstrap.cc:492] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:56.504911 9532 tablet_bootstrap.cc:654] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.504962 31970 tablet_service.cc:1467] Processing CreateTablet for tablet 257cadd9e8ab4f68b700579558af2ed7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.505178 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257cadd9e8ab4f68b700579558af2ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.505540 29582 tablet_service.cc:1467] Processing CreateTablet for tablet a50a38153ff44d1c988838bb39884f20 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.505795 29582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50a38153ff44d1c988838bb39884f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.505950 31840 tablet_service.cc:1467] Processing CreateTablet for tablet a50a38153ff44d1c988838bb39884f20 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.506217 31840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50a38153ff44d1c988838bb39884f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.506378 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 257cadd9e8ab4f68b700579558af2ed7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.506668 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257cadd9e8ab4f68b700579558af2ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.506922 32101 tablet_service.cc:1467] Processing CreateTablet for tablet a50a38153ff44d1c988838bb39884f20 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.507138 32101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50a38153ff44d1c988838bb39884f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.508574 9531 tablet_bootstrap.cc:492] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:00:56.509464 9531 tablet_bootstrap.cc:654] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.509932 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 257cadd9e8ab4f68b700579558af2ed7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=1c111799e1b44e43b3eeed10aa1a5ae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:56.510178 9615 tablet_bootstrap.cc:492] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:00:56.510237 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257cadd9e8ab4f68b700579558af2ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.511909 9615 tablet_bootstrap.cc:654] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.512718 9612 tablet_bootstrap.cc:492] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:56.514950 9612 tablet_bootstrap.cc:654] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.515363 9615 tablet_bootstrap.cc:492] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423: No bootstrap required, opened a new log
I20240502 14:00:56.515499 9615 ts_tablet_manager.cc:1367] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:56.516005 9615 raft_consensus.cc:348] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.517000 9615 raft_consensus.cc:374] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.517315 9615 raft_consensus.cc:729] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.517534 9615 consensus_queue.cc:260] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.517735 9531 tablet_bootstrap.cc:492] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4: No bootstrap required, opened a new log
I20240502 14:00:56.517827 9531 ts_tablet_manager.cc:1367] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:00:56.518015 9615 ts_tablet_manager.cc:1398] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:56.518052 9616 tablet_bootstrap.cc:492] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:00:56.518261 9531 raft_consensus.cc:348] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.518370 9531 raft_consensus.cc:374] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.518404 9531 raft_consensus.cc:729] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.518504 9531 consensus_queue.cc:260] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.518613 9613 tablet_bootstrap.cc:492] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:56.518765 9531 ts_tablet_manager.cc:1398] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:56.523859 9532 tablet_bootstrap.cc:492] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:56.523949 9532 ts_tablet_manager.cc:1367] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:00:56.524420 9532 raft_consensus.cc:348] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.524518 9532 raft_consensus.cc:374] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.524551 9532 raft_consensus.cc:729] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.524658 9532 consensus_queue.cc:260] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.524936 9532 ts_tablet_manager.cc:1398] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:56.524953 9613 tablet_bootstrap.cc:654] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.525090 9532 tablet_bootstrap.cc:492] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Bootstrap starting.
I20240502 14:00:56.525815 9616 tablet_bootstrap.cc:654] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.526010 9532 tablet_bootstrap.cc:654] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.526733 9613 tablet_bootstrap.cc:492] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:56.526816 9613 ts_tablet_manager.cc:1367] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:56.527482 9613 raft_consensus.cc:348] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.527685 9613 raft_consensus.cc:374] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.527751 9613 raft_consensus.cc:729] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.527863 9613 consensus_queue.cc:260] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.529464 9532 tablet_bootstrap.cc:492] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: No bootstrap required, opened a new log
I20240502 14:00:56.529610 9613 ts_tablet_manager.cc:1398] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:56.529909 9613 tablet_bootstrap.cc:492] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1: Bootstrap starting.
I20240502 14:00:56.529911 9532 ts_tablet_manager.cc:1367] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:56.530356 9532 raft_consensus.cc:348] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.530506 9532 raft_consensus.cc:374] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.530591 9532 raft_consensus.cc:729] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.530756 9532 consensus_queue.cc:260] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.531075 9613 tablet_bootstrap.cc:654] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.534767 9532 ts_tablet_manager.cc:1398] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:00:56.535497 9616 tablet_bootstrap.cc:492] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd: No bootstrap required, opened a new log
I20240502 14:00:56.536592 9616 ts_tablet_manager.cc:1367] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:00:56.537830 29451 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:00:56.541117 9616 raft_consensus.cc:348] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.541366 9616 raft_consensus.cc:374] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.541462 9616 raft_consensus.cc:729] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.543998 9612 tablet_bootstrap.cc:492] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:56.544126 9612 ts_tablet_manager.cc:1367] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
I20240502 14:00:56.544637 9612 raft_consensus.cc:348] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.544749 9612 raft_consensus.cc:374] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.544783 9612 raft_consensus.cc:729] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.544903 9612 consensus_queue.cc:260] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.545022 9613 tablet_bootstrap.cc:492] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1: No bootstrap required, opened a new log
I20240502 14:00:56.545150 9613 ts_tablet_manager.cc:1367] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:00:56.545177 9612 ts_tablet_manager.cc:1398] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:56.545413 9612 tablet_bootstrap.cc:492] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:00:56.546478 9613 raft_consensus.cc:348] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.546572 9613 raft_consensus.cc:374] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.546602 9613 raft_consensus.cc:729] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.546717 9613 consensus_queue.cc:260] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.546958 9613 ts_tablet_manager.cc:1398] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:56.549909 9616 consensus_queue.cc:260] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.558458 9612 tablet_bootstrap.cc:654] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:56.562187 9616 ts_tablet_manager.cc:1398] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20240502 14:00:56.565388 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:56.567055 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:56.567318 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:56.572424 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:56.576509 8814 raft_consensus.cc:482] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:56.576601 8814 raft_consensus.cc:504] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.577061 8814 leader_election.cc:288] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:56.578024 29325 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:00:56.581913 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22543bedc3f043bba1d05b675a8f3c75" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:56.582033 31990 raft_consensus.cc:2454] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:56.582412 32056 leader_election.cc:302] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:56.582626 8814 raft_consensus.cc:2781] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:56.582672 8814 raft_consensus.cc:482] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:56.582705 8814 raft_consensus.cc:3037] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.582796 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:56.583860 8814 raft_consensus.cc:504] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.584197 8814 leader_election.cc:288] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:00:56.584694 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22543bedc3f043bba1d05b675a8f3c75" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:56.584806 32253 raft_consensus.cc:3037] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.585834 32253 raft_consensus.cc:2454] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:56.587571 31859 raft_consensus.cc:1261] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:56.587618 32120 raft_consensus.cc:1261] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:56.587954 9471 consensus_queue.cc:1035] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.588070 32058 leader_election.cc:302] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:56.588070 8739 consensus_queue.cc:1035] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:56.588963 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22543bedc3f043bba1d05b675a8f3c75" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:56.589085 31992 raft_consensus.cc:3037] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.589154 9295 raft_consensus.cc:2781] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:56.589293 9295 raft_consensus.cc:686] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:56.589406 9295 consensus_queue.cc:237] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:56.590220 31992 raft_consensus.cc:2454] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:56.596401 29320 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet a4a635382b94471088de77b6c431d334 (table rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2]). Expected version 1 got 0
I20240502 14:00:56.598240 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:56.602031 9640 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:56.602972 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22543bedc3f043bba1d05b675a8f3c75" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:56.603027 9640 tablet.cc:1620] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:56.603119 32255 raft_consensus.cc:2362] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1. Re-sending same reply.
I20240502 14:00:56.605973 9612 tablet_bootstrap.cc:492] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e: No bootstrap required, opened a new log
I20240502 14:00:56.606215 29325 catalog_manager.cc:5576] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995) does not have the latest schema for tablet a4a635382b94471088de77b6c431d334 (table rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2]). Expected version 1 got 0
I20240502 14:00:56.606237 9641 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:56.606525 9641 tablet.cc:1620] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:56.606499 29325 catalog_manager.cc:5484] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:56.606945 9612 ts_tablet_manager.cc:1367] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.002s
I20240502 14:00:56.607364 9534 mvcc.cc:205] Tried to move back new op lower bound from 7023241038178250752 to 7023241037561274368. Current Snapshot: MvccSnapshot[applied={T|T < 7023241038178250752}]
I20240502 14:00:56.610158 32101 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:00:56.612233 9612 raft_consensus.cc:348] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.613593 9612 raft_consensus.cc:374] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:56.613687 9612 raft_consensus.cc:729] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:00:56.614100 9612 consensus_queue.cc:260] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.615152 29320 catalog_manager.cc:5576] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647) does not have the latest schema for tablet a4a635382b94471088de77b6c431d334 (table rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2]). Expected version 1 got 0
I20240502 14:00:56.616475 9612 ts_tablet_manager.cc:1398] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:00:56.623206 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2] alter complete (version 1)
I20240502 14:00:56.620852 9642 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:00:56.623555 9642 tablet.cc:1620] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:56.663424 8814 raft_consensus.cc:482] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:56.663535 8814 raft_consensus.cc:504] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.663900 8814 leader_election.cc:288] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:56.664208 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50a38153ff44d1c988838bb39884f20" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:00:56.664232 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50a38153ff44d1c988838bb39884f20" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:00:56.664343 31859 raft_consensus.cc:2454] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:56.664352 29600 raft_consensus.cc:2454] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 0.
I20240502 14:00:56.664614 32056 leader_election.cc:302] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:56.665025 8814 raft_consensus.cc:2781] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:56.665110 8814 raft_consensus.cc:482] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:56.665179 8814 raft_consensus.cc:3037] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.666237 8814 raft_consensus.cc:504] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.666566 8814 leader_election.cc:288] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:56.667045 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50a38153ff44d1c988838bb39884f20" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:00:56.667155 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50a38153ff44d1c988838bb39884f20" candidate_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:56.667193 29600 raft_consensus.cc:3037] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.667258 31861 raft_consensus.cc:3037] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.668228 29600 raft_consensus.cc:2454] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:56.668269 31861 raft_consensus.cc:2454] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b3fa3988ae4af9ba8cc5ffb79d5bfb in term 1.
I20240502 14:00:56.668536 32056 leader_election.cc:302] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb; no voters:
I20240502 14:00:56.668804 8814 raft_consensus.cc:2781] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:56.668905 8814 raft_consensus.cc:686] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Becoming Leader. State: Replica: f6b3fa3988ae4af9ba8cc5ffb79d5bfb, State: Running, Role: LEADER
I20240502 14:00:56.669027 8814 consensus_queue.cc:237] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:56.671483 29328 catalog_manager.cc:5484] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135). New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:56.682380 31861 consensus_queue.cc:237] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:56.683501 29470 raft_consensus.cc:1261] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:56.683768 29864 raft_consensus.cc:1261] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:56.683890 6614 consensus_queue.cc:1035] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:56.684252 9063 consensus_queue.cc:1035] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:56.685570 6614 raft_consensus.cc:2932] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:56.686134 29470 raft_consensus.cc:2932] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:56.687696 29864 raft_consensus.cc:2932] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:56.689399 29325 catalog_manager.cc:5484] T 631725c869674c45ac79c3f05cee8356 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:56.692319 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 631725c869674c45ac79c3f05cee8356 with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:56.692873 9662 tablet_replica.cc:330] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:56.693080 9662 raft_consensus.cc:2229] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:56.693244 9662 raft_consensus.cc:2258] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:56.693744 9662 ts_tablet_manager.cc:1875] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:56.696439 9662 ts_tablet_manager.cc:1888] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:56.696501 9662 log.cc:1196] T 631725c869674c45ac79c3f05cee8356 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/631725c869674c45ac79c3f05cee8356
I20240502 14:00:56.696975 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 631725c869674c45ac79c3f05cee8356 (table rebalancer_extra_table_60 [id=84138f6ac11f4257afb1119fb4f0a4b3]) successfully deleted
I20240502 14:00:56.726130 29320 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:00:56.731815 9640 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:56.732141 9640 tablet.cc:1620] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: MemRowSet was empty: no flush needed.
I20240502 14:00:56.733537 9642 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:56.733812 9642 tablet.cc:1620] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: MemRowSet was empty: no flush needed.
I20240502 14:00:56.734783 9641 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:00:56.735085 9641 tablet.cc:1620] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: MemRowSet was empty: no flush needed.
I20240502 14:00:56.739562 29328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table [id=d1780fb74c064ee5a515e77838f9a2e2] alter complete (version 2)
I20240502 14:00:56.745159 8448 raft_consensus.cc:482] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:56.745244 8448 raft_consensus.cc:504] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.745615 8448 leader_election.cc:288] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:56.745980 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257cadd9e8ab4f68b700579558af2ed7" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:00:56.746089 32255 raft_consensus.cc:2454] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:56.746410 29406 leader_election.cc:302] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:00:56.746886 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257cadd9e8ab4f68b700579558af2ed7" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:00:56.746999 31990 raft_consensus.cc:2454] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 0.
I20240502 14:00:56.748247 8448 raft_consensus.cc:2781] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:56.748303 8448 raft_consensus.cc:482] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:56.748343 8448 raft_consensus.cc:3037] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.749321 8448 raft_consensus.cc:504] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.749617 8448 leader_election.cc:288] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:00:56.750442 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257cadd9e8ab4f68b700579558af2ed7" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:00:56.750535 32255 raft_consensus.cc:3037] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.754298 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257cadd9e8ab4f68b700579558af2ed7" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:56.754406 31990 raft_consensus.cc:3037] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:56.755347 31990 raft_consensus.cc:2454] T 257cadd9e8ab4f68b700579558af2ed7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:56.755784 29404 leader_election.cc:302] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
I20240502 14:00:56.754833 32255 raft_consensus.cc:2454] T 257cadd9e8ab4f68b700579558af2ed7 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:00:56.755939 8655 raft_consensus.cc:2781] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:56.756028 8655 raft_consensus.cc:686] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:00:56.756152 8655 consensus_queue.cc:237] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:00:56.758266 29320 catalog_manager.cc:5484] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:00:56.843655 29329 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40958:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:00:56.849709 9640 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:56.850663 9641 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:56.854028 9642 tablet.cc:1722] T a4a635382b94471088de77b6c431d334 P f0e01c4f581843e9ad4ced97f0072dc4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:00:56.857993 29321 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 091053291c28421292039b46a8cdf3ff: rebalancer_dynamic_table_new_name [id=d1780fb74c064ee5a515e77838f9a2e2] alter complete (version 3)
I20240502 14:00:56.884848 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:56.886708 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:56.888904 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:56.893921 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:56.895673 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:56.901798 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:56.902232 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:56.908161 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:56.979979 32122 consensus_queue.cc:237] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:56.981526 29600 raft_consensus.cc:1261] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:56.981819 29863 raft_consensus.cc:1261] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:56.981897 9295 consensus_queue.cc:1035] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:56.983171 9295 consensus_queue.cc:1035] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:56.983865 9295 raft_consensus.cc:2932] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:56.985383 29600 raft_consensus.cc:2932] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:56.986253 29863 raft_consensus.cc:2932] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:56.987654 29323 catalog_manager.cc:5484] T 75347e9c867547efaf78355aec6dfb40 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:56.990721 31970 tablet_service.cc:1514] Processing DeleteTablet for tablet 75347e9c867547efaf78355aec6dfb40 with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:56.991081 9734 tablet_replica.cc:330] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:56.991277 9734 raft_consensus.cc:2229] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:56.991688 9734 raft_consensus.cc:2258] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:56.992179 9734 ts_tablet_manager.cc:1875] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:56.994002 9734 ts_tablet_manager.cc:1888] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:56.994069 9734 log.cc:1196] T 75347e9c867547efaf78355aec6dfb40 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/75347e9c867547efaf78355aec6dfb40
I20240502 14:00:56.994598 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 75347e9c867547efaf78355aec6dfb40 (table rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]) successfully deleted
I20240502 14:00:57.083685 8814 consensus_queue.cc:1035] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:57.090023 8814 consensus_queue.cc:1035] T 22543bedc3f043bba1d05b675a8f3c75 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:57.132261 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.132678 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:57.137418 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.145717 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:57.147432 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:57.156343 9637 consensus_queue.cc:1035] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:57.159713 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:57.170745 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.171139 9637 consensus_queue.cc:1035] T a50a38153ff44d1c988838bb39884f20 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:57.171255 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.248186 32120 consensus_queue.cc:237] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:57.249351 29470 raft_consensus.cc:1261] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:57.249352 29732 raft_consensus.cc:1261] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:57.249809 9637 consensus_queue.cc:1035] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:57.249984 8814 consensus_queue.cc:1035] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:57.252050 9637 raft_consensus.cc:2932] T f8b939db0e5b464dbcecbf8e18cccadf P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:57.252406 29732 raft_consensus.cc:2932] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:57.253845 29470 raft_consensus.cc:2932] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:57.257522 29324 catalog_manager.cc:5484] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:57.261953 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet f8b939db0e5b464dbcecbf8e18cccadf with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:57.264034 9817 tablet_replica.cc:330] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:57.264189 9817 raft_consensus.cc:2229] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:57.264312 9817 raft_consensus.cc:2258] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:57.264782 9817 ts_tablet_manager.cc:1875] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:57.267040 9817 ts_tablet_manager.cc:1888] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:57.267138 9817 log.cc:1196] T f8b939db0e5b464dbcecbf8e18cccadf P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/f8b939db0e5b464dbcecbf8e18cccadf
I20240502 14:00:57.267701 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet f8b939db0e5b464dbcecbf8e18cccadf (table rebalancer_extra_table_12 [id=f34c06e2e1fb43b1a0bb0b1f95be7b8e]) successfully deleted
I20240502 14:00:57.330698 8655 consensus_queue.cc:1035] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:57.337859 9855 consensus_queue.cc:1035] T 257cadd9e8ab4f68b700579558af2ed7 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:57.424399 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:57.431034 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.431728 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.433123 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.434592 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:57.443243 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:57.445533 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.454674 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:57.520671 31991 consensus_queue.cc:237] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:57.522099 29732 raft_consensus.cc:1261] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:57.522341 32120 raft_consensus.cc:1261] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:57.522487 8697 consensus_queue.cc:1035] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:57.522743 8404 consensus_queue.cc:1035] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:57.524561 8697 raft_consensus.cc:2932] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:57.524958 32120 raft_consensus.cc:2932] T 0ec7b45e60d140d4a6e937307370fdc9 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:57.525291 29732 raft_consensus.cc:2932] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:57.528777 29328 catalog_manager.cc:5484] T 0ec7b45e60d140d4a6e937307370fdc9 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 2 to 3, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:57.533766 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec7b45e60d140d4a6e937307370fdc9 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:57.534070 9893 tablet_replica.cc:330] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:57.534273 9893 raft_consensus.cc:2229] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:57.534459 9893 raft_consensus.cc:2258] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:57.534972 9893 ts_tablet_manager.cc:1875] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:57.537629 9893 ts_tablet_manager.cc:1888] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:57.537756 9893 log.cc:1196] T 0ec7b45e60d140d4a6e937307370fdc9 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/0ec7b45e60d140d4a6e937307370fdc9
I20240502 14:00:57.538662 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 0ec7b45e60d140d4a6e937307370fdc9 (table rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]) successfully deleted
I20240502 14:00:57.657497 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.658331 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.659919 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:57.662822 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.683626 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:57.688117 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:57.688526 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:57.708076 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.781417 31991 consensus_queue.cc:237] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:57.782476 29470 raft_consensus.cc:1261] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:57.782534 29732 raft_consensus.cc:1261] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:57.782831 9887 consensus_queue.cc:1035] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:57.783178 9887 consensus_queue.cc:1035] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:57.784637 9887 raft_consensus.cc:2932] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:57.785077 29470 raft_consensus.cc:2932] T 824fb738a4b84e13b5fc283a26c92355 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:57.785394 29732 raft_consensus.cc:2932] T 824fb738a4b84e13b5fc283a26c92355 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:57.788357 29326 catalog_manager.cc:5484] T 824fb738a4b84e13b5fc283a26c92355 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:57.792547 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 824fb738a4b84e13b5fc283a26c92355 with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:57.793090 9959 tablet_replica.cc:330] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:57.793269 9959 raft_consensus.cc:2229] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:57.793457 9959 raft_consensus.cc:2258] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:57.793936 9959 ts_tablet_manager.cc:1875] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:57.796365 9959 ts_tablet_manager.cc:1888] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:57.796429 9959 log.cc:1196] T 824fb738a4b84e13b5fc283a26c92355 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/824fb738a4b84e13b5fc283a26c92355
I20240502 14:00:57.796968 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 824fb738a4b84e13b5fc283a26c92355 (table rebalancer_extra_table_30 [id=cdd7601275784592abb3096673da8058]) successfully deleted
I20240502 14:00:57.913470 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.921590 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:57.926951 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.929103 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:57.934371 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:57.936267 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:57.953044 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:57.953372 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:58.027755 32253 consensus_queue.cc:237] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:00:58.028988 32120 raft_consensus.cc:1261] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:58.028990 29732 raft_consensus.cc:1261] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:58.029402 9393 consensus_queue.cc:1035] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:58.029611 10013 consensus_queue.cc:1035] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:58.031692 10013 raft_consensus.cc:2932] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:58.032043 29732 raft_consensus.cc:2932] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:58.035368 29326 catalog_manager.cc:5484] T 39db58bc129046bbbefea02b3eee4243 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:58.038213 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 39db58bc129046bbbefea02b3eee4243 with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43670
I20240502 14:00:58.038812 32120 raft_consensus.cc:2932] T 39db58bc129046bbbefea02b3eee4243 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:00:58.041424 10038 tablet_replica.cc:330] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:00:58.041623 10038 raft_consensus.cc:2229] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:58.041769 10038 raft_consensus.cc:2258] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:58.042137 10038 ts_tablet_manager.cc:1875] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:58.045310 10038 ts_tablet_manager.cc:1888] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:58.045439 10038 log.cc:1196] T 39db58bc129046bbbefea02b3eee4243 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/39db58bc129046bbbefea02b3eee4243
I20240502 14:00:58.045969 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 39db58bc129046bbbefea02b3eee4243 (table rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]) successfully deleted
I20240502 14:00:58.189049 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:58.191633 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:58.196187 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.197341 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:58.204622 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:58.210943 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:58.212558 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.228319 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:58.283984 32120 consensus_queue.cc:237] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:58.285009 29600 raft_consensus.cc:1261] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:58.285049 29470 raft_consensus.cc:1261] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:58.285430 8814 consensus_queue.cc:1035] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:58.285634 10014 consensus_queue.cc:1035] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:58.287375 9637 raft_consensus.cc:2932] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:58.287855 29470 raft_consensus.cc:2932] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:58.287882 29600 raft_consensus.cc:2932] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:58.291033 29321 catalog_manager.cc:5484] T 541806c11f074316a5342050427f4849 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:58.293812 31970 tablet_service.cc:1514] Processing DeleteTablet for tablet 541806c11f074316a5342050427f4849 with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59524
I20240502 14:00:58.294196 10105 tablet_replica.cc:330] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:00:58.294421 10105 raft_consensus.cc:2229] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:58.294576 10105 raft_consensus.cc:2258] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:58.295159 10105 ts_tablet_manager.cc:1875] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:58.297930 10105 ts_tablet_manager.cc:1888] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:58.298059 10105 log.cc:1196] T 541806c11f074316a5342050427f4849 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/541806c11f074316a5342050427f4849
I20240502 14:00:58.298612 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 541806c11f074316a5342050427f4849 (table rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]) successfully deleted
I20240502 14:00:58.440616 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:58.447774 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:58.448832 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:58.448820 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:58.452257 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.453001 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:58.461592 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.495327 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:58.544893 29729 consensus_queue.cc:237] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:00:58.546655 32122 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:58.546909 31861 raft_consensus.cc:1261] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:58.547140 8756 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:58.547353 9539 consensus_queue.cc:1035] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:58.549291 8756 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:58.549564 31861 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:58.553205 32122 raft_consensus.cc:2932] T 463dddd3473a42a68b488e1c465e997f P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:58.553306 29328 catalog_manager.cc:5484] T 463dddd3473a42a68b488e1c465e997f P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:00:58.557463 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 463dddd3473a42a68b488e1c465e997f with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37732
I20240502 14:00:58.557834 10182 tablet_replica.cc:330] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:00:58.557986 10182 raft_consensus.cc:2229] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:58.558167 10182 raft_consensus.cc:2258] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:58.558775 10182 ts_tablet_manager.cc:1875] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:58.561614 10182 ts_tablet_manager.cc:1888] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:00:58.561683 10182 log.cc:1196] T 463dddd3473a42a68b488e1c465e997f P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/463dddd3473a42a68b488e1c465e997f
I20240502 14:00:58.562254 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 463dddd3473a42a68b488e1c465e997f (table rebalance_test_table_5 [id=b2cbaec759914c8f9d9ad41f5a3dbfeb]) successfully deleted
I20240502 14:00:58.677556 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:58.678241 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.693461 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.696071 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:58.699049 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:58.712666 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:58.721824 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:58.724372 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:58.803215 31861 consensus_queue.cc:237] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:00:58.804420 31992 raft_consensus.cc:1261] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:58.804462 32255 raft_consensus.cc:1261] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:58.804777 9063 consensus_queue.cc:1035] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:58.804963 10012 consensus_queue.cc:1035] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:58.806792 10012 raft_consensus.cc:2932] T 3afd58ca519e4bc8a6ab00c37abf9f1a P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:58.807693 31992 raft_consensus.cc:2932] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:58.807803 32255 raft_consensus.cc:2932] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:58.810966 29325 catalog_manager.cc:5484] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:00:58.814654 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet 3afd58ca519e4bc8a6ab00c37abf9f1a with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57986
I20240502 14:00:58.815012 10249 tablet_replica.cc:330] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:00:58.815227 10249 raft_consensus.cc:2229] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:58.815410 10249 raft_consensus.cc:2258] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:58.815919 10249 ts_tablet_manager.cc:1875] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:58.818447 10249 ts_tablet_manager.cc:1888] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:58.818578 10249 log.cc:1196] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/3afd58ca519e4bc8a6ab00c37abf9f1a
I20240502 14:00:58.819259 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 3afd58ca519e4bc8a6ab00c37abf9f1a (table rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]) successfully deleted
I20240502 14:00:58.932026 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:58.932909 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.933339 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:58.956303 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:58.956576 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:58.958381 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:58.960124 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:58.965274 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.043278 29600 consensus_queue.cc:237] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:00:59.044543 31992 raft_consensus.cc:1261] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:59.044642 29470 raft_consensus.cc:1261] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:59.045017 10319 consensus_queue.cc:1035] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:59.045248 8779 consensus_queue.cc:1035] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:59.047158 10319 raft_consensus.cc:2932] T 4a2ab852e31d4181a8597b9047a08554 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:59.047483 29470 raft_consensus.cc:2932] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:59.050963 29325 catalog_manager.cc:5484] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:59.052058 31992 raft_consensus.cc:2932] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:00:59.054960 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a2ab852e31d4181a8597b9047a08554 with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:00:59.056453 10322 tablet_replica.cc:330] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:00:59.056630 10322 raft_consensus.cc:2229] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:59.056809 10322 raft_consensus.cc:2258] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:59.057304 10322 ts_tablet_manager.cc:1875] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:59.059881 10322 ts_tablet_manager.cc:1888] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:59.060016 10322 log.cc:1196] T 4a2ab852e31d4181a8597b9047a08554 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/4a2ab852e31d4181a8597b9047a08554
I20240502 14:00:59.060642 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 4a2ab852e31d4181a8597b9047a08554 (table rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]) successfully deleted
I20240502 14:00:59.204174 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:59.205837 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:59.210915 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.214167 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.216737 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.220273 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:59.230806 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.232810 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:59.304857 31991 consensus_queue.cc:237] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:00:59.306062 29600 raft_consensus.cc:1261] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:59.306082 31859 raft_consensus.cc:1261] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:59.306416 8697 consensus_queue.cc:1035] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:59.306607 9887 consensus_queue.cc:1035] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:59.308297 9887 raft_consensus.cc:2932] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:59.311075 31859 raft_consensus.cc:2932] T 768df43846d94951a3a48f6f626bb473 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:59.311847 29321 catalog_manager.cc:5484] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:59.315418 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 768df43846d94951a3a48f6f626bb473 with delete_type TABLET_DATA_TOMBSTONED (TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57612
I20240502 14:00:59.327405 10393 tablet_replica.cc:330] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:00:59.327569 10393 raft_consensus.cc:2229] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:59.327714 10393 raft_consensus.cc:2258] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:59.328087 10393 ts_tablet_manager.cc:1875] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:59.331094 10393 ts_tablet_manager.cc:1888] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:00:59.331199 10393 log.cc:1196] T 768df43846d94951a3a48f6f626bb473 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/768df43846d94951a3a48f6f626bb473
I20240502 14:00:59.331121 29600 raft_consensus.cc:2932] T 768df43846d94951a3a48f6f626bb473 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:00:59.331754 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 768df43846d94951a3a48f6f626bb473 (table rebalancer_extra_table_28 [id=d98d7a29328444e29b6f089725ccc3f9]) successfully deleted
I20240502 14:00:59.448835 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.451272 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:59.460685 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.461179 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:59.463902 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.474211 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.489850 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:59.513553 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:59.549501 29729 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6"
dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40818
I20240502 14:00:59.549685 29729 raft_consensus.cc:595] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:59.672403 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.673084 29843 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:00:59.675036 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:59.676156 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:00:59.679895 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.688488 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.716356 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:59.719823 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.734162 9539 raft_consensus.cc:979] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: : Instructing follower 4aebe6d7a3f34bf7b2bbbfbef1077463 to start an election
I20240502 14:00:59.734261 9539 raft_consensus.cc:1067] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Signalling peer 4aebe6d7a3f34bf7b2bbbfbef1077463 to start an election
I20240502 14:00:59.734903 29864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6"
dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
from {username='slave'} at 127.22.55.131:50041
I20240502 14:00:59.735044 29864 raft_consensus.cc:482] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:59.735095 29864 raft_consensus.cc:3037] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:59.736106 29864 raft_consensus.cc:504] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:59.736594 29864 leader_election.cc:288] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:00:59.737038 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:00:59.737151 31992 raft_consensus.cc:3037] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:59.737325 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:00:59.737442 31861 raft_consensus.cc:3037] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:59.738126 31992 raft_consensus.cc:2454] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:00:59.738435 31861 raft_consensus.cc:2454] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:00:59.738958 29797 leader_election.cc:302] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:00:59.739140 10487 raft_consensus.cc:2781] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:59.739209 10487 raft_consensus.cc:686] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:00:59.739341 10487 consensus_queue.cc:237] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:59.740898 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdf925a9c4d456fb3b4ef448a2154c6" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:00:59.741001 29729 raft_consensus.cc:3032] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:59.741039 29729 raft_consensus.cc:729] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:00:59.741176 29729 consensus_queue.cc:260] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:00:59.741374 29729 raft_consensus.cc:3037] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:59.742283 29729 raft_consensus.cc:2454] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:00:59.743683 29323 catalog_manager.cc:5484] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:00:59.909744 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:00:59.914165 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.918366 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.922153 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.925915 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:00:59.931025 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:00:59.931803 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:00:59.942060 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:00.018262 29864 consensus_queue.cc:237] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:00.019500 29602 raft_consensus.cc:1261] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.019652 31859 raft_consensus.cc:1261] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.019877 10487 consensus_queue.cc:1035] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.020241 10487 consensus_queue.cc:1035] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.022094 9471 raft_consensus.cc:2932] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:00.022581 29602 raft_consensus.cc:2932] T a0001d3f124c42eba6129836209a41fa P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:00.022904 31859 raft_consensus.cc:2932] T a0001d3f124c42eba6129836209a41fa P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:00.025555 29321 catalog_manager.cc:5484] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:00.028157 31970 tablet_service.cc:1514] Processing DeleteTablet for tablet a0001d3f124c42eba6129836209a41fa with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59524
I20240502 14:01:00.028522 10581 tablet_replica.cc:330] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:01:00.028688 10581 raft_consensus.cc:2229] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:00.028820 10581 raft_consensus.cc:2258] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:00.029254 10581 ts_tablet_manager.cc:1875] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:00.032011 10581 ts_tablet_manager.cc:1888] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:00.032120 10581 log.cc:1196] T a0001d3f124c42eba6129836209a41fa P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/a0001d3f124c42eba6129836209a41fa
I20240502 14:01:00.032668 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet a0001d3f124c42eba6129836209a41fa (table rebalancer_extra_table_58 [id=88b8a1332acb4bb39b989263a0666a4d]) successfully deleted
I20240502 14:01:00.146785 29732 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:00.147308 10487 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:00.185500 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.187036 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:00.191738 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.194190 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:00.202553 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:00.211625 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.220656 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.235059 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:01:00.250418 31992 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:00.250859 10487 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:00.254918 31859 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:00.257911 9471 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:00.286962 31992 consensus_queue.cc:237] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:00.288162 29864 raft_consensus.cc:1261] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.288242 31859 raft_consensus.cc:1261] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.288544 8697 consensus_queue.cc:1035] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.288741 10644 consensus_queue.cc:1035] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.290477 9887 raft_consensus.cc:2932] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.290908 29864 raft_consensus.cc:2932] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.292096 31859 raft_consensus.cc:2932] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.293922 29329 catalog_manager.cc:5484] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.297797 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 with delete_type TABLET_DATA_TOMBSTONED (TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57612
I20240502 14:01:00.298525 10663 tablet_replica.cc:330] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:01:00.298691 10663 raft_consensus.cc:2229] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:00.298813 10663 raft_consensus.cc:2258] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:00.299454 10663 ts_tablet_manager.cc:1875] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:00.301712 10663 ts_tablet_manager.cc:1888] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:00.301812 10663 log.cc:1196] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/d40f25aa6bcc46fca8cd67a6fd4b86b6
I20240502 14:01:00.302340 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet d40f25aa6bcc46fca8cd67a6fd4b86b6 (table rebalancer_extra_table_42 [id=ccd477487f1048c9a1fc183b892feeb9]) successfully deleted
I20240502 14:01:00.439273 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.445514 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.446923 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:00.451887 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:00.453503 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.471014 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.471596 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:01:00.481698 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:00.543449 31991 consensus_queue.cc:237] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:00.544914 29471 raft_consensus.cc:1261] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.545109 29602 raft_consensus.cc:1261] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.545394 10644 consensus_queue.cc:1035] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.545571 8697 consensus_queue.cc:1035] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.547287 10644 raft_consensus.cc:2932] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:00.549291 29602 raft_consensus.cc:2932] T 1203440714974cb5981e67c2856fb0c7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:00.550568 29471 raft_consensus.cc:2932] T 1203440714974cb5981e67c2856fb0c7 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:00.551928 29327 catalog_manager.cc:5484] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:00.554771 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 1203440714974cb5981e67c2856fb0c7 with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:01:00.555094 10735 tablet_replica.cc:330] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:01:00.555265 10735 raft_consensus.cc:2229] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:00.555397 10735 raft_consensus.cc:2258] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:00.555944 10735 ts_tablet_manager.cc:1875] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:00.558004 10735 ts_tablet_manager.cc:1888] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:00.558077 10735 log.cc:1196] T 1203440714974cb5981e67c2856fb0c7 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/1203440714974cb5981e67c2856fb0c7
I20240502 14:01:00.558571 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 1203440714974cb5981e67c2856fb0c7 (table rebalancer_extra_table_38 [id=90f408924f994699acbbd0b13c215988]) successfully deleted
I20240502 14:01:00.876642 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.877140 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.877930 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:00.879329 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.880869 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:00.907737 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:00.914991 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:00.927619 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:01:00.978912 32255 consensus_queue.cc:237] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:00.980170 29864 raft_consensus.cc:1261] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.980295 29733 raft_consensus.cc:1261] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:00.980638 10551 consensus_queue.cc:1035] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.980840 10013 consensus_queue.cc:1035] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:00.982661 10013 raft_consensus.cc:2932] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.985874 29863 raft_consensus.cc:2932] T 5ae0eeb814e34729b6284d838d458ddf P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.986413 29733 raft_consensus.cc:2932] T 5ae0eeb814e34729b6284d838d458ddf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.986553 29322 catalog_manager.cc:5484] T 5ae0eeb814e34729b6284d838d458ddf P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:00.990805 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae0eeb814e34729b6284d838d458ddf with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43670
I20240502 14:01:00.991330 10813 tablet_replica.cc:330] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:01:00.991531 10813 raft_consensus.cc:2229] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:00.991704 10813 raft_consensus.cc:2258] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:00.992149 10813 ts_tablet_manager.cc:1875] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:00.994740 10813 ts_tablet_manager.cc:1888] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:00.994817 10813 log.cc:1196] T 5ae0eeb814e34729b6284d838d458ddf P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/5ae0eeb814e34729b6284d838d458ddf
I20240502 14:01:00.995371 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 5ae0eeb814e34729b6284d838d458ddf (table rebalancer_extra_table_18 [id=6bdc85bb28b549df95dc82f989e03c2f]) successfully deleted
I20240502 14:01:01.129468 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.136433 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.137226 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:01.143038 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:01.148926 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:01.150615 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:01:01.153048 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.175238 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.238976 32253 consensus_queue.cc:237] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:01.240103 29602 raft_consensus.cc:1261] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:01.240137 31991 raft_consensus.cc:1261] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:01.240425 10551 consensus_queue.cc:1035] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:01.240614 10013 consensus_queue.cc:1035] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:01.242439 10013 raft_consensus.cc:2932] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.242902 29602 raft_consensus.cc:2932] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.242905 31991 raft_consensus.cc:2932] T 1ea3465ad91540cea8906bcc406efd07 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.246214 29329 catalog_manager.cc:5484] T 1ea3465ad91540cea8906bcc406efd07 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 2 to 3, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) evicted. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.251195 32101 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea3465ad91540cea8906bcc406efd07 with delete_type TABLET_DATA_TOMBSTONED (TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57612
I20240502 14:01:01.251551 10885 tablet_replica.cc:330] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: stopping tablet replica
I20240502 14:01:01.251742 10885 raft_consensus.cc:2229] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:01.251866 10885 raft_consensus.cc:2258] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:01.252321 10885 ts_tablet_manager.cc:1875] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:01.255187 10885 ts_tablet_manager.cc:1888] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:01.255264 10885 log.cc:1196] T 1ea3465ad91540cea8906bcc406efd07 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-6/wal/wals/1ea3465ad91540cea8906bcc406efd07
I20240502 14:01:01.255829 29313 catalog_manager.cc:4830] TS f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): tablet 1ea3465ad91540cea8906bcc406efd07 (table rebalancer_extra_table_16 [id=f73e834c7f884616a60231a0e06fc84c]) successfully deleted
I20240502 14:01:01.399715 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:01.400436 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.401635 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:01:01.402140 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.402362 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.404805 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:01.405050 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.419749 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:01.490420 29733 consensus_queue.cc:237] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:01.491729 32120 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:01.492084 10636 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:01.493703 10747 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.495512 32120 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.497709 31990 raft_consensus.cc:1261] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:01.497712 29326 catalog_manager.cc:5484] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 6 to 8, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 2 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.500738 10636 consensus_queue.cc:1035] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:01.501714 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet be8afc0891fb43fa9ffc863ad2bc90d2 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37732
I20240502 14:01:01.502041 31990 raft_consensus.cc:2932] T be8afc0891fb43fa9ffc863ad2bc90d2 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.507027 10954 tablet_replica.cc:330] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:01:01.507248 10954 raft_consensus.cc:2229] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:01.507577 10954 raft_consensus.cc:2258] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:01.508507 10954 ts_tablet_manager.cc:1875] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:01.510836 10954 ts_tablet_manager.cc:1888] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:01:01.510937 10954 log.cc:1196] T be8afc0891fb43fa9ffc863ad2bc90d2 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/be8afc0891fb43fa9ffc863ad2bc90d2
I20240502 14:01:01.511554 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet be8afc0891fb43fa9ffc863ad2bc90d2 (table rebalance_test_table_19 [id=006c128f55b0472d8dbb7a02b6ead43c]) successfully deleted
I20240502 14:01:01.731762 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:01.740869 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:01.747220 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.750818 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.752307 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.762420 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:01.768661 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:01.795903 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:01:01.846647 29864 consensus_queue.cc:237] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:01.847872 31990 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:01.847887 31859 raft_consensus.cc:1261] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:01.848227 9471 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:01.848402 11058 consensus_queue.cc:1035] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:01.850152 11058 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:01.850570 31990 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:01.850726 31859 raft_consensus.cc:2932] T 9bdf925a9c4d456fb3b4ef448a2154c6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:01.853741 29326 catalog_manager.cc:5484] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:01.857965 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bdf925a9c4d456fb3b4ef448a2154c6 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54432
I20240502 14:01:01.858317 11092 tablet_replica.cc:330] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:01:01.858470 11092 raft_consensus.cc:2229] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:01.858687 11092 raft_consensus.cc:2258] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:01.859315 11092 ts_tablet_manager.cc:1875] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:01.861660 11092 ts_tablet_manager.cc:1888] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:01:01.861723 11092 log.cc:1196] T 9bdf925a9c4d456fb3b4ef448a2154c6 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/9bdf925a9c4d456fb3b4ef448a2154c6
I20240502 14:01:01.862306 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 9bdf925a9c4d456fb3b4ef448a2154c6 (table rebalance_test_table_13 [id=9f9308c8174d4d6691d4621f18cee869]) successfully deleted
I20240502 14:01:01.891913 29600 consensus_queue.cc:237] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:01.893869 32255 raft_consensus.cc:1261] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:01.893895 29864 raft_consensus.cc:1261] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:01.893990 29536 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e69badccfbc416f92d982cf394ea0cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:01.894291 11120 consensus_queue.cc:1035] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:01.894490 11121 consensus_queue.cc:1035] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:01.896144 11121 raft_consensus.cc:2932] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:01.896528 29863 raft_consensus.cc:2932] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:01.899597 32255 raft_consensus.cc:2932] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:01.900427 29321 catalog_manager.cc:5484] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:01.928439 29733 consensus_queue.cc:237] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:01.929746 32255 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:01.929879 29471 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:01.930235 10636 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:01.930457 10953 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:01:01.930958 29666 consensus_peers.cc:485] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92cc4394003840a5b1d59b0fe72c5164. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:01.933061 10953 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.933491 32255 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.935321 29471 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.936842 29326 catalog_manager.cc:5484] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:01.959842 31859 consensus_queue.cc:237] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:01.961489 29863 raft_consensus.cc:1261] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:01.961621 32120 raft_consensus.cc:1261] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:01.962035 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3062c7a80b284f6c8cb28f8c53c9c0a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:01.962203 10012 consensus_queue.cc:1035] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:01.962508 10012 consensus_queue.cc:1035] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:01.964584 11159 raft_consensus.cc:2932] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.965005 29863 raft_consensus.cc:2932] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.968073 29321 catalog_manager.cc:5484] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.968950 32120 raft_consensus.cc:2932] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:01.989910 31990 consensus_queue.cc:237] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:01.991425 32255 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:01:01.991544 29733 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:01:01.991829 11180 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:01:01.992033 31926 consensus_peers.cc:485] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1116d9f165b488796ab466f9a866471. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:01.992247 11180 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:01.995958 8697 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:01.996387 29733 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:01.998109 32255 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:02.000655 29326 catalog_manager.cc:5484] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 4 to 5, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:02.020241 31990 consensus_queue.cc:237] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.021709 32255 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:02.021898 29471 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:02.022322 31926 consensus_peers.cc:485] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec8e4ac081604ec8a3f463f740d9119c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.022380 11180 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:02.022583 8697 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:02.024606 11180 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.025067 32255 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.028009 29323 catalog_manager.cc:5484] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 3 to 4, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.029564 29471 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.067332 29600 consensus_queue.cc:237] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:02.069083 31990 raft_consensus.cc:1261] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.069186 29863 raft_consensus.cc:1261] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:02.069367 29535 consensus_peers.cc:485] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 995449d84e4b4d4692f8f225b5a44830. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.069554 11121 consensus_queue.cc:1035] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.069883 9609 consensus_queue.cc:1035] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.072494 11121 raft_consensus.cc:2932] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.074282 31990 raft_consensus.cc:2932] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.075122 29863 raft_consensus.cc:2932] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.076530 29321 catalog_manager.cc:5484] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 2, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.096963 32255 consensus_queue.cc:237] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.098425 32120 raft_consensus.cc:1261] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.098815 10013 consensus_queue.cc:1035] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:02.099009 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04f0f9efcbb544d7b8292158bf769426. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.099035 31990 raft_consensus.cc:1261] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.099435 11207 consensus_queue.cc:1035] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.101301 10805 raft_consensus.cc:2932] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.101809 32120 raft_consensus.cc:2932] T 04f0f9efcbb544d7b8292158bf769426 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.103261 31990 raft_consensus.cc:2932] T 04f0f9efcbb544d7b8292158bf769426 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.105554 29323 catalog_manager.cc:5484] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.132324 32120 consensus_queue.cc:237] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
W20240502 14:01:02.134176 32057 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5733940f27f345d8a314a9e51d2a455e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.134302 31990 raft_consensus.cc:1261] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.134941 32255 raft_consensus.cc:1261] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.135083 11256 consensus_queue.cc:1035] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.135802 10302 consensus_queue.cc:1035] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.137342 9637 raft_consensus.cc:2932] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.137740 32255 raft_consensus.cc:2932] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.140093 31990 raft_consensus.cc:2932] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.140794 29321 catalog_manager.cc:5484] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.163007 32120 consensus_queue.cc:237] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:02.164489 32255 raft_consensus.cc:1261] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.164554 29471 raft_consensus.cc:1261] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:02.164733 32057 consensus_peers.cc:485] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.165035 11255 consensus_queue.cc:1035] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.165268 11256 consensus_queue.cc:1035] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.167307 11255 raft_consensus.cc:2932] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:02.167738 29471 raft_consensus.cc:2932] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:02.171341 32255 raft_consensus.cc:2932] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:02.171801 29326 catalog_manager.cc:5484] T b23fc8a56f054a5b8ec0272ad99a93ea P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:02.198555 32255 consensus_queue.cc:237] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
W20240502 14:01:02.200508 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acdcc4d7e5754103a08d76bdd714a68e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.200655 31859 raft_consensus.cc:1261] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.201056 10013 consensus_queue.cc:1035] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.203378 31990 raft_consensus.cc:1261] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.204159 11207 consensus_queue.cc:1035] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.206233 11207 raft_consensus.cc:2932] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.206696 31991 raft_consensus.cc:2932] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.208757 31859 raft_consensus.cc:2932] T acdcc4d7e5754103a08d76bdd714a68e P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.210772 29326 catalog_manager.cc:5484] T acdcc4d7e5754103a08d76bdd714a68e P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.230360 32120 consensus_queue.cc:237] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.231701 31859 raft_consensus.cc:1261] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.231783 31991 raft_consensus.cc:1261] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:02.232023 32056 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09f77f3b6bf14a46b2e41c91125f6bd7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.232215 10302 consensus_queue.cc:1035] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.232393 11256 consensus_queue.cc:1035] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.234004 10302 raft_consensus.cc:2932] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.234462 31859 raft_consensus.cc:2932] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.236119 31991 raft_consensus.cc:2932] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.238276 29321 catalog_manager.cc:5484] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:02.261128 31859 consensus_queue.cc:237] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.262573 32255 raft_consensus.cc:1261] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:02.262724 31795 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eab1144457d34c25897c5b7a691409fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.262825 32120 raft_consensus.cc:1261] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.263124 10012 consensus_queue.cc:1035] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.263309 11162 consensus_queue.cc:1035] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.265025 10012 raft_consensus.cc:2932] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.265450 32255 raft_consensus.cc:2932] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.267078 32120 raft_consensus.cc:2932] T eab1144457d34c25897c5b7a691409fe P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.269081 29326 catalog_manager.cc:5484] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:02.290760 32120 consensus_queue.cc:237] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:02.292323 32255 raft_consensus.cc:1261] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:02.292948 32058 consensus_peers.cc:485] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.293251 9637 consensus_queue.cc:1035] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.293324 29864 raft_consensus.cc:1261] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.293663 9637 consensus_queue.cc:1035] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.295325 11256 raft_consensus.cc:2932] T 31df5c53f5004ef399a1bd3458f57573 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:02.295775 29864 raft_consensus.cc:2932] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:02.297164 32255 raft_consensus.cc:2932] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:02.298938 29323 catalog_manager.cc:5484] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:02.324054 29602 consensus_queue.cc:237] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:02.325547 31859 raft_consensus.cc:1261] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:02.325547 31991 raft_consensus.cc:1261] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:02.325781 29535 consensus_peers.cc:485] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ece4fc38b254afeae58936444fe047d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.325973 11120 consensus_queue.cc:1035] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.326138 11121 consensus_queue.cc:1035] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:02.327821 11121 raft_consensus.cc:2932] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.328222 31859 raft_consensus.cc:2932] T 7ece4fc38b254afeae58936444fe047d P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.330016 31991 raft_consensus.cc:2932] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.331406 29321 catalog_manager.cc:5484] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:02.368325 29602 consensus_queue.cc:237] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:02.369760 29471 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:02.369745 32120 raft_consensus.cc:1261] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:01:02.369904 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253ea421888249d180928ba0aef28ddd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.370172 10850 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:02.370354 11120 consensus_queue.cc:1035] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:02.372536 11120 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.372912 32120 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.374614 29470 raft_consensus.cc:2932] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.377316 29323 catalog_manager.cc:5484] T 253ea421888249d180928ba0aef28ddd P f6b3fa3988ae4af9ba8cc5ffb79d5bfb reported cstate change: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:02.423882 11413 ts_tablet_manager.cc:902] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:02.424471 11413 tablet_copy_client.cc:308] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:01:02.425351 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet a1116d9f165b488796ab466f9a866471 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:44929)
I20240502 14:01:02.425458 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet a1116d9f165b488796ab466f9a866471 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:44929: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-a1116d9f165b488796ab466f9a866471
I20240502 14:01:02.426484 32012 tablet_copy_source_session.cc:205] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.428452 11413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116d9f165b488796ab466f9a866471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.432906 11413 tablet_copy_client.cc:783] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.434252 11413 tablet_copy_client.cc:647] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.435806 11413 tablet_copy_client.cc:523] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.438380 11413 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:01:02.445348 11413 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.445729 11413 tablet_bootstrap.cc:492] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:01:02.445931 11413 ts_tablet_manager.cc:1367] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:01:02.446373 11413 raft_consensus.cc:348] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:02.446501 11413 raft_consensus.cc:729] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.446666 11413 consensus_queue.cc:260] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:02.447064 11413 ts_tablet_manager.cc:1398] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:02.447571 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-a1116d9f165b488796ab466f9a866471 received from {username='slave'} at 127.22.55.132:44929
I20240502 14:01:02.447644 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-a1116d9f165b488796ab466f9a866471 on tablet a1116d9f165b488796ab466f9a866471 with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:01:02.474275 11433 ts_tablet_manager.cc:902] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:02.474828 11433 tablet_copy_client.cc:308] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:01:02.475589 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 4e69badccfbc416f92d982cf394ea0cd from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:34633)
I20240502 14:01:02.475692 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 4e69badccfbc416f92d982cf394ea0cd from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:34633: session id = efdbf7bdfe0b41bdadb1add053b09a7e-4e69badccfbc416f92d982cf394ea0cd
I20240502 14:01:02.476590 29622 tablet_copy_source_session.cc:205] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.480417 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69badccfbc416f92d982cf394ea0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.482486 11433 tablet_copy_client.cc:783] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.482631 11433 tablet_copy_client.cc:647] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.483640 11433 tablet_copy_client.cc:523] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.484809 11433 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:01:02.489758 11433 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.490033 11433 tablet_bootstrap.cc:492] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:01:02.490186 11433 ts_tablet_manager.cc:1367] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240502 14:01:02.490643 11433 raft_consensus.cc:348] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.490753 11433 raft_consensus.cc:729] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.490913 11433 consensus_queue.cc:260] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.491241 11433 ts_tablet_manager.cc:1398] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:02.493666 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-4e69badccfbc416f92d982cf394ea0cd received from {username='slave'} at 127.22.55.131:34633
I20240502 14:01:02.493746 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-4e69badccfbc416f92d982cf394ea0cd on tablet 4e69badccfbc416f92d982cf394ea0cd with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:01:02.507812 11439 ts_tablet_manager.cc:902] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:02.508543 11439 tablet_copy_client.cc:308] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:01:02.509099 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet 04f0f9efcbb544d7b8292158bf769426 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:39449)
I20240502 14:01:02.509202 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet 04f0f9efcbb544d7b8292158bf769426 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:39449: session id = d6e2e2c10def4331bec17bd110c9b423-04f0f9efcbb544d7b8292158bf769426
I20240502 14:01:02.510301 32275 tablet_copy_source_session.cc:205] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.512120 11439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f0f9efcbb544d7b8292158bf769426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.514710 11445 ts_tablet_manager.cc:902] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:02.515636 11445 tablet_copy_client.cc:308] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:01:02.515719 11439 tablet_copy_client.cc:783] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.515868 11439 tablet_copy_client.cc:647] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.516183 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 92cc4394003840a5b1d59b0fe72c5164 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:50019)
I20240502 14:01:02.516290 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 92cc4394003840a5b1d59b0fe72c5164 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:50019: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-92cc4394003840a5b1d59b0fe72c5164
I20240502 14:01:02.517278 29753 tablet_copy_source_session.cc:205] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.518229 11445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cc4394003840a5b1d59b0fe72c5164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.516048 11450 ts_tablet_manager.cc:902] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:02.520560 11450 tablet_copy_client.cc:308] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:01:02.521167 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:42227)
I20240502 14:01:02.521270 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet 3062c7a80b284f6c8cb28f8c53c9c0a4 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:42227: session id = d6e2e2c10def4331bec17bd110c9b423-3062c7a80b284f6c8cb28f8c53c9c0a4
I20240502 14:01:02.522249 31865 tablet_copy_source_session.cc:205] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.522699 11445 tablet_copy_client.cc:783] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.522917 11445 tablet_copy_client.cc:647] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.524211 11445 tablet_copy_client.cc:523] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.524562 11439 tablet_copy_client.cc:523] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.525903 11439 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:01:02.526134 11445 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:01:02.528512 11450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3062c7a80b284f6c8cb28f8c53c9c0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.532008 11450 tablet_copy_client.cc:783] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.532215 11450 tablet_copy_client.cc:647] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.533461 11439 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.533713 11439 tablet_bootstrap.cc:492] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:01:02.533862 11439 ts_tablet_manager.cc:1367] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 14:01:02.534168 11450 tablet_copy_client.cc:523] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.534268 11439 raft_consensus.cc:348] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.534382 11439 raft_consensus.cc:729] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.534487 11439 consensus_queue.cc:260] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.534878 11439 ts_tablet_manager.cc:1398] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:02.535405 11450 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:01:02.536165 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-04f0f9efcbb544d7b8292158bf769426 received from {username='slave'} at 127.22.55.130:39449
I20240502 14:01:02.536245 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-04f0f9efcbb544d7b8292158bf769426 on tablet 04f0f9efcbb544d7b8292158bf769426 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:02.544327 11450 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.544592 11450 tablet_bootstrap.cc:492] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:01:02.544777 11450 ts_tablet_manager.cc:1367] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 14:01:02.545264 11450 raft_consensus.cc:348] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.545377 11450 raft_consensus.cc:729] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.545485 11450 consensus_queue.cc:260] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.545742 11450 ts_tablet_manager.cc:1398] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:02.546509 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-3062c7a80b284f6c8cb28f8c53c9c0a4 received from {username='slave'} at 127.22.55.130:42227
I20240502 14:01:02.546583 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-3062c7a80b284f6c8cb28f8c53c9c0a4 on tablet 3062c7a80b284f6c8cb28f8c53c9c0a4 with peer d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:02.555963 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.559226 11433 ts_tablet_manager.cc:902] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:02.559799 11433 tablet_copy_client.cc:308] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:01:02.560983 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet ec8e4ac081604ec8a3f463f740d9119c from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:46293)
I20240502 14:01:02.561089 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet ec8e4ac081604ec8a3f463f740d9119c from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:46293: session id = efdbf7bdfe0b41bdadb1add053b09a7e-ec8e4ac081604ec8a3f463f740d9119c
I20240502 14:01:02.561946 32012 tablet_copy_source_session.cc:205] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.564621 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8e4ac081604ec8a3f463f740d9119c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.569087 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.569341 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.571906 11445 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.572188 11445 tablet_bootstrap.cc:492] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:01:02.572379 11445 ts_tablet_manager.cc:1367] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.005s
I20240502 14:01:02.572844 11445 raft_consensus.cc:348] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:02.572968 11445 raft_consensus.cc:729] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.573096 11445 consensus_queue.cc:260] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:02.573732 11445 ts_tablet_manager.cc:1398] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:02.578740 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-92cc4394003840a5b1d59b0fe72c5164 received from {username='slave'} at 127.22.55.134:50019
I20240502 14:01:02.578821 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-92cc4394003840a5b1d59b0fe72c5164 on tablet 92cc4394003840a5b1d59b0fe72c5164 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:01:02.583416 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:02.584538 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:02.590497 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:02.591089 11433 tablet_copy_client.cc:783] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.591286 11433 tablet_copy_client.cc:647] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.595007 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.602515 32101 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240502 14:01:02.615825 11478 ts_tablet_manager.cc:902] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:02.627216 11433 tablet_copy_client.cc:523] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.627799 11478 tablet_copy_client.cc:308] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:01:02.628346 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 5733940f27f345d8a314a9e51d2a455e from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:33761)
I20240502 14:01:02.628460 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 5733940f27f345d8a314a9e51d2a455e from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:33761: session id = efdbf7bdfe0b41bdadb1add053b09a7e-5733940f27f345d8a314a9e51d2a455e
I20240502 14:01:02.628973 11433 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:01:02.629390 32143 tablet_copy_source_session.cc:205] T 5733940f27f345d8a314a9e51d2a455e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.634032 11478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5733940f27f345d8a314a9e51d2a455e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.636180 11478 tablet_copy_client.cc:783] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.636310 11478 tablet_copy_client.cc:647] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.637959 11478 tablet_copy_client.cc:523] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.638084 11450 ts_tablet_manager.cc:902] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:02.638520 11450 tablet_copy_client.cc:308] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.135:40995
I20240502 14:01:02.639451 11478 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:01:02.639592 32143 tablet_copy_service.cc:140] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Received BeginTabletCopySession request for tablet 09f77f3b6bf14a46b2e41c91125f6bd7 from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:35209)
I20240502 14:01:02.639685 32143 tablet_copy_service.cc:161] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Beginning new tablet copy session on tablet 09f77f3b6bf14a46b2e41c91125f6bd7 from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:35209: session id = d6e2e2c10def4331bec17bd110c9b423-09f77f3b6bf14a46b2e41c91125f6bd7
I20240502 14:01:02.640318 32143 tablet_copy_source_session.cc:205] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.641861 11450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f77f3b6bf14a46b2e41c91125f6bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.644169 11450 tablet_copy_client.cc:783] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.644323 11450 tablet_copy_client.cc:647] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.644719 11478 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.644981 11478 tablet_bootstrap.cc:492] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:01:02.645148 11478 ts_tablet_manager.cc:1367] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 14:01:02.645613 11478 raft_consensus.cc:348] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.645732 11478 raft_consensus.cc:729] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.645848 11478 consensus_queue.cc:260] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.646176 11478 ts_tablet_manager.cc:1398] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:02.646587 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-5733940f27f345d8a314a9e51d2a455e received from {username='slave'} at 127.22.55.131:33761
I20240502 14:01:02.646657 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-5733940f27f345d8a314a9e51d2a455e on tablet 5733940f27f345d8a314a9e51d2a455e with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:01:02.648804 11433 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.649097 11433 tablet_bootstrap.cc:492] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:01:02.649259 11433 ts_tablet_manager.cc:1367] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 14:01:02.649672 11433 raft_consensus.cc:348] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.649789 11433 raft_consensus.cc:729] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.650266 11433 consensus_queue.cc:260] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.651312 11433 ts_tablet_manager.cc:1398] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:02.651940 11489 ts_tablet_manager.cc:902] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:02.652626 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-ec8e4ac081604ec8a3f463f740d9119c received from {username='slave'} at 127.22.55.131:46293
I20240502 14:01:02.652704 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-ec8e4ac081604ec8a3f463f740d9119c on tablet ec8e4ac081604ec8a3f463f740d9119c with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:01:02.652714 11489 tablet_copy_client.cc:308] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:01:02.653270 11450 tablet_copy_client.cc:523] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.653350 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 995449d84e4b4d4692f8f225b5a44830 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:49783)
I20240502 14:01:02.653445 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 995449d84e4b4d4692f8f225b5a44830 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:49783: session id = f0e01c4f581843e9ad4ced97f0072dc4-995449d84e4b4d4692f8f225b5a44830
I20240502 14:01:02.654263 29622 tablet_copy_source_session.cc:205] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.654505 11450 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:01:02.655056 11489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995449d84e4b4d4692f8f225b5a44830. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.657497 11489 tablet_copy_client.cc:783] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.657649 11489 tablet_copy_client.cc:647] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.658795 11450 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.658867 11495 ts_tablet_manager.cc:902] T acdcc4d7e5754103a08d76bdd714a68e P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:02.659123 11450 tablet_bootstrap.cc:492] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
F20240502 14:01:02.659027 11495 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0867c6a22c2c4efd9a9babff371cc8ba.queue_time_us, tablet-download-pool-0867c6a22c2c4efd9a9babff371cc8ba.queue_time_us) (tablet-download-pool-0867c6a22c2c4efd9a9babff371cc8ba.run_cpu_time_us, tablet-download-pool-0867c6a22c2c4efd9a9babff371cc8ba.run_cpu_time_us) (tablet-download-pool-0867c6a22c2c4efd9a9babff371cc8ba.run_wall_time_us, tablet-download-pool-0867c6a22c2c4efd9a9babff371cc8ba.run_wall_time_us) (tablet-download-pool-09ce1f1d33dd4beca86a484a6f498175.queue_time_us, tablet-download-pool-09ce1f1d33dd4beca86a484a6f498175.queue_time_us) (tablet-download-pool-09ce1f1d33dd4beca86a484a6f498175.run_cpu_time_us, tablet-download-pool-09ce1f1d33dd4beca86a484a6f498175.run_cpu_time_us) (tablet-download-pool-09ce1f1d33dd4beca86a484a6f498175.run_wall_time_us, tablet-download-pool-09ce1f1d33dd4beca86a484a6f498175.run_wall_time_us) (tablet-download-pool-0ec7b45e60d140d4a6e937307370fdc9.queue_time_us, tablet-download-pool-0ec7b45e60d140d4a6e937307370fdc9.queue_time_us) (tablet-download-pool-0ec7b45e60d140d4a6e937307370fdc9.run_cpu_time_us, tablet-download-pool-0ec7b45e60d140d4a6e937307370fdc9.run_cpu_time_us) (tablet-download-pool-0ec7b45e60d140d4a6e937307370fdc9.run_wall_time_us, tablet-download-pool-0ec7b45e60d140d4a6e937307370fdc9.run_wall_time_us) (tablet-download-pool-253ea421888249d180928ba0aef28ddd.queue_time_us, tablet-download-pool-253ea421888249d180928ba0aef28ddd.queue_time_us) (tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_cpu_time_us, tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_cpu_time_us) (tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_wall_time_us, tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_wall_time_us) (tablet-download-pool-3d2ed5a4845b4bd78b19af6380c77982.queue_time_us, tablet-download-pool-3d2ed5a4845b4bd78b19af6380c77982.queue_time_us) (tablet-download-pool-3d2ed5a4845b4bd78b19af6380c77982.run_cpu_time_us, tablet-download-pool-3d2ed5a4845b4bd78b19af6380c77982.run_cpu_time_us) (tablet-download-pool-3d2ed5a4845b4bd78b19af6380c77982.run_wall_time_us, tablet-download-pool-3d2ed5a4845b4bd78b19af6380c77982.run_wall_time_us) (tablet-download-pool-463dddd3473a42a68b488e1c465e997f.queue_time_us, tablet-download-pool-463dddd3473a42a68b488e1c465e997f.queue_time_us) (tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_cpu_time_us, tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_cpu_time_us) (tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_wall_time_us, tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_wall_time_us) (tablet-download-pool-59f4faff46704ca1837edd139596d9e2.queue_time_us, tablet-download-pool-59f4faff46704ca1837edd139596d9e2.queue_time_us) (tablet-download-pool-59f4faff46704ca1837edd139596d9e2.run_cpu_time_us, tablet-download-pool-59f4faff46704ca1837edd139596d9e2.run_cpu_time_us) (tablet-download-pool-59f4faff46704ca1837edd139596d9e2.run_wall_time_us, tablet-download-pool-59f4faff46704ca1837edd139596d9e2.run_wall_time_us) (tablet-download-pool-7650009dd8954f11ab2ecafef765692c.queue_time_us, tablet-download-pool-7650009dd8954f11ab2ecafef765692c.queue_time_us) (tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_cpu_time_us, tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_cpu_time_us) (tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_wall_time_us, tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_wall_time_us) (tablet-download-pool-7ebf61381a92430caa0c38cc04d14cb6.queue_time_us, tablet-download-pool-7ebf61381a92430caa0c38cc04d14cb6.queue_time_us) (tablet-download-pool-7ebf61381a92430caa0c38cc04d14cb6.run_cpu_time_us, tablet-download-pool-7ebf61381a92430caa0c38cc04d14cb6.run_cpu_time_us) (tablet-download-pool-7ebf61381a92430caa0c38cc04d14cb6.run_wall_time_us, tablet-download-pool-7ebf61381a92430caa0c38cc04d14cb6.run_wall_time_us) (tablet-download-pool-93e77c40b5674865ba6ccce626545ec7.queue_time_us, tablet-download-pool-93e77c40b5674865ba6ccce626545ec7.queue_time_us) (tablet-download-pool-93e77c40b5674865ba6ccce626545ec7.run_cpu_time_us, tablet-download-pool-93e77c40b5674865ba6ccce626545ec7.run_cpu_time_us) (tablet-download-pool-93e77c40b5674865ba6ccce626545ec7.run_wall_time_us, tablet-download-pool-93e77c40b5674865ba6ccce626545ec7.run_wall_time_us) (tablet-download-pool-9790e7ee6d8945bf85f129ef3f9900ca.queue_time_us, tablet-download-pool-9790e7ee6d8945bf85f129ef3f9900ca.queue_time_us) (tablet-download-pool-9790e7ee6d8945bf85f129ef3f9900ca.run_cpu_time_us, tablet-download-pool-9790e7ee6d8945bf85f129ef3f9900ca.run_cpu_time_us) (tablet-download-pool-9790e7ee6d8945bf85f129ef3f9900ca.run_wall_time_us, tablet-download-pool-9790e7ee6d8945bf85f129ef3f9900ca.run_wall_time_us) (tablet-download-pool-a3bf63852df44cf1b762b7a981428c97.queue_time_us, tablet-download-pool-a3bf63852df44cf1b762b7a981428c97.queue_time_us) (tablet-download-pool-a3bf63852df44cf1b762b7a981428c97.run_cpu_time_us, tablet-download-pool-a3bf63852df44cf1b762b7a981428c97.run_cpu_time_us) (tablet-download-pool-a3bf63852df44cf1b762b7a981428c97.run_wall_time_us, tablet-download-pool-a3bf63852df44cf1b762b7a981428c97.run_wall_time_us) (tablet-download-pool-acdcc4d7e5754103a08d76bdd714a68e.queue_time_us, tablet-download-pool-acdcc4d7e5754103a08d76bdd714a68e.queue_time_us) (tablet-download-pool-bd91ed1ae1354a84a0ff668efdbc03d9.queue_time_us, tablet-download-pool-bd91ed1ae1354a84a0ff668efdbc03d9.queue_time_us) (tablet-download-pool-bd91ed1ae1354a84a0ff668efdbc03d9.run_cpu_time_us, tablet-download-pool-bd91ed1ae1354a84a0ff668efdbc03d9.run_cpu_time_us) (tablet-download-pool-bd91ed1ae1354a84a0ff668efdbc03d9.run_wall_time_us, tablet-download-pool-bd91ed1ae1354a84a0ff668efdbc03d9.run_wall_time_us) (tablet-download-pool-be8afc0891fb43fa9ffc863ad2bc90d2.queue_time_us, tablet-download-pool-be8afc0891fb43fa9ffc863ad2bc90d2.queue_time_us) (tablet-download-pool-be8afc0891fb43fa9ffc863ad2bc90d2.run_cpu_time_us, tablet-download-pool-be8afc0891fb43fa9ffc863ad2bc90d2.run_cpu_time_us) (tablet-download-pool-be8afc0891fb43fa9ffc863ad2bc90d2.run_wall_time_us, tablet-download-pool-be8afc0891fb43fa9ffc863ad2bc90d2.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 14:01:02.659353 11450 ts_tablet_manager.cc:1367] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
*** Check failure stack trace: ***
I20240502 14:01:02.659826 11450 raft_consensus.cc:348] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.659921 11450 raft_consensus.cc:729] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.660066 11450 consensus_queue.cc:260] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:02.660372 11450 ts_tablet_manager.cc:1398] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:02.660828 32143 tablet_copy_service.cc:342] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-09f77f3b6bf14a46b2e41c91125f6bd7 received from {username='slave'} at 127.22.55.130:35209
I20240502 14:01:02.660888 32143 tablet_copy_service.cc:433] P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-09f77f3b6bf14a46b2e41c91125f6bd7 on tablet 09f77f3b6bf14a46b2e41c91125f6bd7 with peer d6e2e2c10def4331bec17bd110c9b423
@ 0x7faba1947dcd google::LogMessage::Fail() at ??:0
I20240502 14:01:02.664950 11433 ts_tablet_manager.cc:902] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:02.665463 11433 tablet_copy_client.cc:308] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:01:02.665903 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet eab1144457d34c25897c5b7a691409fe from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:35643)
I20240502 14:01:02.665993 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet eab1144457d34c25897c5b7a691409fe from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:35643: session id = efdbf7bdfe0b41bdadb1add053b09a7e-eab1144457d34c25897c5b7a691409fe
I20240502 14:01:02.667279 11489 tablet_copy_client.cc:523] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.667487 31865 tablet_copy_source_session.cc:205] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.668560 11489 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:01:02.669406 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab1144457d34c25897c5b7a691409fe. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7faba194bb93 google::LogMessage::SendToLog() at ??:0
I20240502 14:01:02.672299 11489 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.672381 11433 tablet_copy_client.cc:783] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.672513 11433 tablet_copy_client.cc:647] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.672539 11489 tablet_bootstrap.cc:492] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:01:02.673030 11489 ts_tablet_manager.cc:1367] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:01:02.673676 11489 raft_consensus.cc:348] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:02.673784 11489 raft_consensus.cc:729] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.673925 11489 consensus_queue.cc:260] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:02.674317 11489 ts_tablet_manager.cc:1398] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:02.674734 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-995449d84e4b4d4692f8f225b5a44830 received from {username='slave'} at 127.22.55.133:49783
I20240502 14:01:02.674800 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-995449d84e4b4d4692f8f225b5a44830 on tablet 995449d84e4b4d4692f8f225b5a44830 with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:02.676438 11433 tablet_copy_client.cc:523] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7faba19477cc google::LogMessage::Flush() at ??:0
I20240502 14:01:02.678033 11433 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
@ 0x7faba1948f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faba2db1044 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:01:02.685408 11433 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.685650 11433 tablet_bootstrap.cc:492] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:01:02.685782 11433 ts_tablet_manager.cc:1367] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240502 14:01:02.686311 11433 raft_consensus.cc:348] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.686501 11433 raft_consensus.cc:729] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.686633 11433 consensus_queue.cc:260] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:02.687292 11433 ts_tablet_manager.cc:1398] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7faba2d95f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:01:02.691632 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-eab1144457d34c25897c5b7a691409fe received from {username='slave'} at 127.22.55.131:35643
I20240502 14:01:02.691704 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-eab1144457d34c25897c5b7a691409fe on tablet eab1144457d34c25897c5b7a691409fe with peer efdbf7bdfe0b41bdadb1add053b09a7e
@ 0x7faba2d93efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7faba594677a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7faba594fa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7faba59d076f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7faba59e44f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7faba59cf226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7faba59dec0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c0a918075c std::function<>::operator()() at ??:0
@ 0x7faba2d98e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faba2d996ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7faba2d9b03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c0a918075c std::function<>::operator()() at ??:0
@ 0x7faba2d88807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faba35c66db start_thread at ??:0
I20240502 14:01:02.740056 11510 ts_tablet_manager.cc:902] T 7ece4fc38b254afeae58936444fe047d P f6b3fa3988ae4af9ba8cc5ffb79d5bfb: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
@ 0x7faba0afe71f clone at ??:0
I20240502 14:01:02.802482 11489 ts_tablet_manager.cc:902] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:02.802983 11489 tablet_copy_client.cc:308] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:01:02.803524 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 253ea421888249d180928ba0aef28ddd from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:49783)
I20240502 14:01:02.803633 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 253ea421888249d180928ba0aef28ddd from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:49783: session id = f0e01c4f581843e9ad4ced97f0072dc4-253ea421888249d180928ba0aef28ddd
I20240502 14:01:02.804648 29622 tablet_copy_source_session.cc:205] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.805704 11489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253ea421888249d180928ba0aef28ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.808229 11489 tablet_copy_client.cc:783] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.808373 11489 tablet_copy_client.cc:647] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.812492 11489 tablet_copy_client.cc:523] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.814189 11489 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:01:02.851307 11489 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:02.851625 11489 tablet_bootstrap.cc:492] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:01:02.851809 11489 ts_tablet_manager.cc:1367] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.004s
I20240502 14:01:02.852349 11489 raft_consensus.cc:348] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:02.852474 11489 raft_consensus.cc:729] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:01:02.852623 11489 consensus_queue.cc:260] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:02.853003 11489 ts_tablet_manager.cc:1398] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:02.853475 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-253ea421888249d180928ba0aef28ddd received from {username='slave'} at 127.22.55.133:49783
I20240502 14:01:02.853545 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-253ea421888249d180928ba0aef28ddd on tablet 253ea421888249d180928ba0aef28ddd with peer f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:02.909066 29864 raft_consensus.cc:1203] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240502 14:01:02.913304 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.918684 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.922744 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.925470 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:02.927347 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:02.943321 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:02.943987 29732 raft_consensus.cc:1203] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:02.959757 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:02.976389 29732 raft_consensus.cc:1203] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:01:03.013515 31991 raft_consensus.cc:1203] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:03.074774 29600 raft_consensus.cc:1203] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:01:03.080695 31859 raft_consensus.cc:1203] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:01:03.092381 29666 connection.cc:535] client connection to 127.22.55.135:40995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.092530 29404 connection.cc:535] client connection to 127.22.55.135:40995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.092551 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.092661 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.093233 29535 connection.cc:535] client connection to 127.22.55.135:40995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.093345 29535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.093699 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.093762 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.093829 31925 connection.cc:535] client connection to 127.22.55.135:40995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.093919 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.094182 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.094240 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.094394 31794 connection.cc:535] client connection to 127.22.55.135:40995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.094496 31794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.094682 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.094748 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.094959 29797 connection.cc:698] client connection to 127.22.55.135:40995 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240502 14:01:03.095077 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: sendmsg error: Connection reset by peer (error 104)
W20240502 14:01:03.095113 32188 connection.cc:535] client connection to 127.22.55.135:40995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.095235 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:03.096007 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.096083 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.096148 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.096580 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:03.096658 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.096714 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.096805 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.096864 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.096912 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.097419 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.098647 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.098706 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:03.098735 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.098765 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.098795 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.098971 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.099026 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.099088 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.099136 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.099478 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.099548 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.099598 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.099647 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.103751 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:03.113835 29600 raft_consensus.cc:1203] T 04f0f9efcbb544d7b8292158bf769426 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:01:03.116379 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.127310 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.134464 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.137199 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:03.178665 29732 raft_consensus.cc:1203] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:01:03.189316 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:03.231010 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:03.254976 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.256873 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.259392 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.260262 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:03.263639 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:03.273010 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.274051 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 14:01:03.278826 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.293118 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:03.305769 31859 raft_consensus.cc:1203] T 253ea421888249d180928ba0aef28ddd P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:03.472497 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:03.482357 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:03.484594 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:03.485200 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.494236 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.506973 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.513947 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.682878 11058 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.682996 11058 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:03.683458 11058 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:03.684557 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75347e9c867547efaf78355aec6dfb40" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
W20240502 14:01:03.687253 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.687330 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.687462 9471 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.691510 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:03.691699 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.696305 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.709698 32233 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.712838 29713 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.716253 29582 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:03.739010 8697 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.739137 8697 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:03.739579 8697 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:03.740489 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:01:03.740618 29602 raft_consensus.cc:2454] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:01:03.740985 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
W20240502 14:01:03.742496 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.742561 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.743039 8697 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.743820 10850 raft_consensus.cc:482] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.743909 10850 raft_consensus.cc:504] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.744232 10850 leader_election.cc:288] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:03.745520 29535 leader_election.cc:334] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.745807 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2868541b54c447219db948c92b9a132b" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:03.746044 10805 raft_consensus.cc:482] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.746125 10805 raft_consensus.cc:504] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:03.746207 29537 leader_election.cc:302] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.746451 10850 raft_consensus.cc:2726] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.746469 10805 leader_election.cc:288] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:03.747165 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cda992fe784d9f9015e6a3d431f1cd" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
W20240502 14:01:03.747572 32188 leader_election.cc:334] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.747640 32188 leader_election.cc:302] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.747819 10805 raft_consensus.cc:2726] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.750797 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:03.756306 11575 raft_consensus.cc:482] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.756420 11575 raft_consensus.cc:504] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.756742 11575 leader_election.cc:288] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:03.759002 29666 leader_election.cc:334] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.759217 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65229e977b8543639fb27bbbc3b5e797" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:03.759599 29666 leader_election.cc:302] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.759848 11444 raft_consensus.cc:2726] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.779809 10012 raft_consensus.cc:482] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.779939 10012 raft_consensus.cc:504] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:03.780352 10012 leader_election.cc:288] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:03.781050 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50a38153ff44d1c988838bb39884f20" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
W20240502 14:01:03.781378 31794 leader_election.cc:334] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.781543 31794 leader_election.cc:302] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.781747 10012 raft_consensus.cc:2726] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.784806 11444 raft_consensus.cc:482] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.784902 11444 raft_consensus.cc:504] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.785282 11444 leader_election.cc:288] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.785773 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d6f5e9b84267afa5e019f6bb8c08" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:01:03.786338 29666 leader_election.cc:334] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.786621 29668 leader_election.cc:302] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.787091 11444 raft_consensus.cc:2726] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.790167 10012 raft_consensus.cc:482] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.790256 10012 raft_consensus.cc:504] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:03.790593 10012 leader_election.cc:288] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:03.791932 29600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63de5b3f35e4486859c9361925689d9" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:01:03.792323 10850 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.792418 10850 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:03.792866 10850 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
W20240502 14:01:03.793157 31794 leader_election.cc:334] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.793241 31794 leader_election.cc:302] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.793429 10012 raft_consensus.cc:2726] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.793520 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75347e9c867547efaf78355aec6dfb40" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:03.793663 29863 raft_consensus.cc:2454] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:01:03.794261 29536 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters:
I20240502 14:01:03.794835 10850 raft_consensus.cc:2781] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:03.794916 10850 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.794960 10850 raft_consensus.cc:3037] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:03.795080 29535 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.796168 10850 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:03.796530 10850 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:03.797098 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75347e9c867547efaf78355aec6dfb40" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:01:03.797225 29863 raft_consensus.cc:3037] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:03.797523 29535 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.798225 29863 raft_consensus.cc:2454] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 2.
I20240502 14:01:03.798509 29536 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.798651 10850 raft_consensus.cc:2781] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:03.798745 10850 raft_consensus.cc:686] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [term 2 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:01:03.798923 10850 consensus_queue.cc:237] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:03.801410 29600 consensus_queue.cc:237] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:03.802546 29863 raft_consensus.cc:1261] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:03.802572 31861 raft_consensus.cc:1261] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:03.802948 11121 consensus_queue.cc:1035] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:03.803134 11775 consensus_queue.cc:1035] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:03.805126 10850 raft_consensus.cc:2932] T 995449d84e4b4d4692f8f225b5a44830 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:03.805132 29322 catalog_manager.cc:5484] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 2 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:03.807312 10805 raft_consensus.cc:482] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.807410 10805 raft_consensus.cc:504] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.807757 10805 leader_election.cc:288] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.807873 29863 raft_consensus.cc:2932] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:03.808573 31861 raft_consensus.cc:2932] T 995449d84e4b4d4692f8f225b5a44830 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
W20240502 14:01:03.811621 32188 leader_election.cc:334] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.812451 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2868541b54c447219db948c92b9a132b" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:01:03.812570 29602 raft_consensus.cc:2454] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:03.812930 32188 leader_election.cc:302] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.813158 10805 raft_consensus.cc:2781] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:03.813217 10805 raft_consensus.cc:482] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.813277 10805 raft_consensus.cc:3037] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:03.814314 10805 raft_consensus.cc:504] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.814651 10805 leader_election.cc:288] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.814821 29327 catalog_manager.cc:5484] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 2 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:03.815541 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2868541b54c447219db948c92b9a132b" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:01:03.815635 29602 raft_consensus.cc:3037] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:03.816401 32188 leader_election.cc:334] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.816622 29602 raft_consensus.cc:2454] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:03.816875 32188 leader_election.cc:302] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.817030 10805 raft_consensus.cc:2781] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:03.817116 10805 raft_consensus.cc:686] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:03.817261 10805 consensus_queue.cc:237] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.818387 31970 tablet_service.cc:1514] Processing DeleteTablet for tablet 995449d84e4b4d4692f8f225b5a44830 with delete_type TABLET_DATA_TOMBSTONED (TS 269b366a09ff435ea8ec8dc01d0f9f6e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:59524
I20240502 14:01:03.818884 11790 tablet_replica.cc:330] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: stopping tablet replica
I20240502 14:01:03.819139 11790 raft_consensus.cc:2229] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:03.819326 11790 raft_consensus.cc:2258] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:03.819865 11790 ts_tablet_manager.cc:1875] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:03.820652 29327 catalog_manager.cc:5484] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:03.822328 11790 ts_tablet_manager.cc:1888] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:03.822414 11790 log.cc:1196] T 995449d84e4b4d4692f8f225b5a44830 P 269b366a09ff435ea8ec8dc01d0f9f6e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-5/wal/wals/995449d84e4b4d4692f8f225b5a44830
I20240502 14:01:03.822993 29313 catalog_manager.cc:4830] TS 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): tablet 995449d84e4b4d4692f8f225b5a44830 (table rebalancer_extra_table_20 [id=54a1a1ca436f42aa903e410dd084e04d]) successfully deleted
W20240502 14:01:03.823489 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:03.825366 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.839689 11575 raft_consensus.cc:482] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.839789 11575 raft_consensus.cc:504] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:03.840262 11575 leader_election.cc:288] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.840500 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198b0a675f184a268bf714cf778408a4" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
W20240502 14:01:03.841807 29666 leader_election.cc:334] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.841873 29666 leader_election.cc:302] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.842015 11575 raft_consensus.cc:2726] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.849957 10981 raft_consensus.cc:482] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.850080 10981 raft_consensus.cc:504] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:03.850548 10981 leader_election.cc:288] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
W20240502 14:01:03.851588 29404 leader_election.cc:334] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.851729 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70367c7c718d4d9195f0af7e69bdf2ec" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:03.852083 29406 leader_election.cc:302] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.852293 10981 raft_consensus.cc:2726] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.864900 10012 raft_consensus.cc:482] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.865003 10012 raft_consensus.cc:504] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:03.865371 10012 leader_election.cc:288] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:03.866298 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3a84c1d004412db381854147330dfc" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
W20240502 14:01:03.866760 31794 leader_election.cc:334] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.867087 31794 leader_election.cc:302] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.867327 10012 raft_consensus.cc:2726] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.874373 10981 raft_consensus.cc:482] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.874460 10981 raft_consensus.cc:504] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:03.874835 10981 leader_election.cc:288] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:03.875046 10981 raft_consensus.cc:482] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.875124 10981 raft_consensus.cc:504] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:03.875442 10981 leader_election.cc:288] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:03.875437 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b939db0e5b464dbcecbf8e18cccadf" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:03.875813 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c08999e4e7f4d0bb6a03199ee676f76" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:03.876426 8697 raft_consensus.cc:482] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.876528 8697 raft_consensus.cc:504] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:03.876888 8697 leader_election.cc:288] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.877278 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22543bedc3f043bba1d05b675a8f3c75" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:01:03.878019 29404 leader_election.cc:334] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.878090 29404 leader_election.cc:302] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:03.879036 31925 leader_election.cc:334] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.879102 31925 leader_election.cc:302] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.879309 8697 raft_consensus.cc:2726] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:03.879433 29404 leader_election.cc:334] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.879492 29404 leader_election.cc:302] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: efdbf7bdfe0b41bdadb1add053b09a7e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.879805 10981 raft_consensus.cc:2726] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.879910 10981 raft_consensus.cc:2726] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.880841 10012 raft_consensus.cc:482] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.880910 10805 raft_consensus.cc:482] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.880946 10012 raft_consensus.cc:504] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.881091 10805 raft_consensus.cc:504] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.881326 10012 leader_election.cc:288] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.881567 10805 leader_election.cc:288] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.881785 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5364d50da4cfb8fbbf21913324bc6" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:03.881920 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d6f5e9b84267afa5e019f6bb8c08" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:03.882042 29732 raft_consensus.cc:2454] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:03.882308 32189 leader_election.cc:302] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:03.882457 10805 raft_consensus.cc:2781] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:03.882498 10805 raft_consensus.cc:482] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.882534 10805 raft_consensus.cc:3037] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:03.882665 31794 leader_election.cc:334] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.882730 31794 leader_election.cc:302] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0e01c4f581843e9ad4ced97f0072dc4; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.882894 10012 raft_consensus.cc:2726] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.883720 10805 raft_consensus.cc:504] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.884065 10805 leader_election.cc:288] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:03.884196 32188 leader_election.cc:334] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:03.884691 32188 leader_election.cc:334] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.884856 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e9d6f5e9b84267afa5e019f6bb8c08" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:03.884963 29732 raft_consensus.cc:3037] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:03.885833 29732 raft_consensus.cc:2454] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:03.886234 32189 leader_election.cc:302] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.886399 10805 raft_consensus.cc:2781] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:03.886485 10805 raft_consensus.cc:686] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:03.886618 10805 consensus_queue.cc:237] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.889011 29328 catalog_manager.cc:5484] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:03.928792 11575 raft_consensus.cc:482] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:03.928922 11575 raft_consensus.cc:504] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:03.929363 11575 leader_election.cc:288] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:03.929641 31991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:03.929764 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:03.930781 29666 leader_election.cc:302] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1
I20240502 14:01:03.930960 11575 raft_consensus.cc:2726] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:03.931159 29666 leader_election.cc:334] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.941371 10013 raft_consensus.cc:482] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.941462 10013 raft_consensus.cc:504] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.941864 10013 leader_election.cc:288] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.944087 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5364d50da4cfb8fbbf21913324bc6" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:01:03.944231 31861 raft_consensus.cc:2454] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:03.945187 32188 leader_election.cc:302] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:03.945322 10013 raft_consensus.cc:2781] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:03.945381 10013 raft_consensus.cc:482] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.945426 10013 raft_consensus.cc:3037] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:03.945328 32188 leader_election.cc:334] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.946542 10013 raft_consensus.cc:504] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.947276 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5364d50da4cfb8fbbf21913324bc6" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:01:03.947391 31861 raft_consensus.cc:3037] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:03.948403 31861 raft_consensus.cc:2454] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:03.948633 10013 leader_election.cc:288] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.949927 32188 leader_election.cc:302] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:03.950076 10013 raft_consensus.cc:2781] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:03.950167 10013 raft_consensus.cc:686] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:03.950299 10013 consensus_queue.cc:237] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.954406 29451 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:01:03.955974 9471 raft_consensus.cc:482] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.956077 9471 raft_consensus.cc:504] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:03.956478 9471 leader_election.cc:288] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.956918 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c8871730c4e2ba9f1320381159331" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:03.959681 31970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.960386 29324 catalog_manager.cc:5484] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
W20240502 14:01:03.961647 32188 leader_election.cc:334] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.963949 32233 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:03.968986 29797 leader_election.cc:334] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.969064 29797 leader_election.cc:302] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:03.970953 9471 raft_consensus.cc:2726] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:03.973704 11575 raft_consensus.cc:482] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.973795 11575 raft_consensus.cc:504] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:03.974153 11575 leader_election.cc:288] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.974536 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b939db0e5b464dbcecbf8e18cccadf" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:03.974668 29469 raft_consensus.cc:2454] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:01:03.975067 29668 leader_election.cc:302] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:03.975286 11575 raft_consensus.cc:2781] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:03.975346 11575 raft_consensus.cc:482] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:03.975396 11575 raft_consensus.cc:3037] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:03.975462 29666 leader_election.cc:334] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.976202 29843 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:01:03.976603 11575 raft_consensus.cc:504] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:03.976893 11575 leader_election.cc:288] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:03.978374 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:03.980206 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b939db0e5b464dbcecbf8e18cccadf" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:03.980312 29469 raft_consensus.cc:3037] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:03.981282 29469 raft_consensus.cc:2454] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
I20240502 14:01:03.981600 29668 leader_election.cc:302] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
W20240502 14:01:03.981762 29666 leader_election.cc:334] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:03.982790 11444 raft_consensus.cc:2781] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:03.982985 11444 raft_consensus.cc:686] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:01:03.983163 11444 consensus_queue.cc:237] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:03.985942 29324 catalog_manager.cc:5484] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 2 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:03.994690 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:04.014782 29582 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:04.022982 9471 raft_consensus.cc:482] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.023120 9471 raft_consensus.cc:504] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.023497 9471 leader_election.cc:288] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.024880 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab77cba6cf431e89141d4154c41a9b" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
W20240502 14:01:04.026422 29797 leader_election.cc:334] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.026504 29797 leader_election.cc:302] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.026721 9471 raft_consensus.cc:2726] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.029868 11832 raft_consensus.cc:482] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.029986 11832 raft_consensus.cc:504] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.030406 11832 leader_election.cc:288] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:04.030896 11477 raft_consensus.cc:482] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.030997 11477 raft_consensus.cc:504] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.031358 11477 leader_election.cc:288] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:04.031620 29404 leader_election.cc:334] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.031930 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df051496f954569983892bfe828f653" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:01:04.033680 31925 leader_election.cc:334] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.033748 31925 leader_election.cc:302] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.033948 8697 raft_consensus.cc:2726] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.034157 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a34a1381cf84af6b560101d95447e6d" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:04.034556 29406 leader_election.cc:302] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.034736 11832 raft_consensus.cc:2726] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.039546 11779 raft_consensus.cc:482] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.039647 11779 raft_consensus.cc:504] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.040023 11779 leader_election.cc:288] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.040482 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3a84c1d004412db381854147330dfc" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:01:04.040596 31861 raft_consensus.cc:2454] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:01:04.041535 29535 leader_election.cc:302] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:04.041908 11779 raft_consensus.cc:2781] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.041962 11779 raft_consensus.cc:482] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.042006 11779 raft_consensus.cc:3037] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.043207 11779 raft_consensus.cc:504] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.043546 11779 leader_election.cc:288] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:04.049068 29535 leader_election.cc:334] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.049968 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d3a84c1d004412db381854147330dfc" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:01:04.050093 31861 raft_consensus.cc:3037] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.051292 31861 raft_consensus.cc:2454] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 2.
I20240502 14:01:04.051796 29535 leader_election.cc:302] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:04.052269 11121 raft_consensus.cc:2781] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.052357 11121 raft_consensus.cc:686] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [term 2 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:01:04.052490 11121 consensus_queue.cc:237] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.054121 10013 raft_consensus.cc:482] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
W20240502 14:01:04.054158 29535 leader_election.cc:334] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.054215 10013 raft_consensus.cc:504] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.054607 10013 leader_election.cc:288] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:04.056102 29324 catalog_manager.cc:5484] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 2 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:04.057700 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70367c7c718d4d9195f0af7e69bdf2ec" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:04.057839 29469 raft_consensus.cc:2454] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:04.058216 32190 leader_election.cc:302] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:04.058399 10013 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.058480 10013 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:04.058724 10805 raft_consensus.cc:2781] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.058794 10805 raft_consensus.cc:482] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.058910 10013 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:04.059238 10805 raft_consensus.cc:3037] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.059538 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
I20240502 14:01:04.059631 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:04.059965 32190 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463
I20240502 14:01:04.060249 10805 raft_consensus.cc:504] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.060386 10013 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:04.060516 32188 leader_election.cc:334] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.060576 10805 leader_election.cc:288] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
W20240502 14:01:04.060628 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.060881 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70367c7c718d4d9195f0af7e69bdf2ec" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:04.060974 29469 raft_consensus.cc:3037] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.062052 29469 raft_consensus.cc:2454] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:04.062305 32190 leader_election.cc:302] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:04.062466 10805 raft_consensus.cc:2781] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.062558 10805 raft_consensus.cc:686] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:04.062687 10805 consensus_queue.cc:237] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
W20240502 14:01:04.062901 32188 leader_election.cc:334] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.065308 29324 catalog_manager.cc:5484] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:04.067147 11846 raft_consensus.cc:482] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.067239 11846 raft_consensus.cc:504] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.067585 11846 leader_election.cc:288] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.068143 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b415bd7b51fc4e4a8c6fa6083d989bfe" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
W20240502 14:01:04.068918 29666 leader_election.cc:334] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.069031 29667 leader_election.cc:302] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.069289 11846 raft_consensus.cc:2726] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.080224 10013 raft_consensus.cc:482] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.080374 10013 raft_consensus.cc:504] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.080806 10013 leader_election.cc:288] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:04.081084 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:04.081272 29733 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:04.081971 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
W20240502 14:01:04.082563 32188 leader_election.cc:334] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.082727 32188 leader_election.cc:302] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.082921 10013 raft_consensus.cc:2726] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.090015 31990 consensus_queue.cc:237] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.091254 32253 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:04.091421 29733 raft_consensus.cc:1261] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:04.091625 11477 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:04.091827 10805 raft_consensus.cc:482] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.091818 8697 consensus_queue.cc:1035] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:04.091899 10805 raft_consensus.cc:504] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.092200 10805 leader_election.cc:288] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
W20240502 14:01:04.093199 32188 leader_election.cc:334] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.093335 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df051496f954569983892bfe828f653" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:04.093436 31990 raft_consensus.cc:2454] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:04.093592 8697 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:04.094111 32253 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:04.094533 32188 leader_election.cc:302] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.094717 10013 raft_consensus.cc:2781] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.094767 10013 raft_consensus.cc:482] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.094807 10013 raft_consensus.cc:3037] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.095829 10013 raft_consensus.cc:504] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.096171 10013 leader_election.cc:288] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:04.096835 29324 catalog_manager.cc:5484] T ec8e4ac081604ec8a3f463f740d9119c P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:04.098698 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df051496f954569983892bfe828f653" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:04.098798 31990 raft_consensus.cc:3037] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.099151 29733 raft_consensus.cc:2932] T ec8e4ac081604ec8a3f463f740d9119c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:04.100203 31990 raft_consensus.cc:2454] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:04.102591 11832 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.102689 11832 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:04.102798 32188 leader_election.cc:302] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:04.103173 11477 raft_consensus.cc:482] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.103199 11832 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:04.103257 11477 raft_consensus.cc:504] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.103420 10013 raft_consensus.cc:2781] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.103502 10013 raft_consensus.cc:686] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:04.103567 11477 leader_election.cc:288] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.103618 10013 consensus_queue.cc:237] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.104323 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:04.104444 32253 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:04.105326 31925 leader_election.cc:334] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:04.105381 32188 leader_election.cc:334] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.105464 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8e4ac081604ec8a3f463f740d9119c with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57986
I20240502 14:01:04.105981 11869 tablet_replica.cc:330] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:01:04.106324 11869 raft_consensus.cc:2229] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:04.106568 11869 raft_consensus.cc:2258] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 14:01:04.106906 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
I20240502 14:01:04.107432 11869 ts_tablet_manager.cc:1875] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:04.107470 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab77cba6cf431e89141d4154c41a9b" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:04.107590 29863 raft_consensus.cc:2454] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:04.107792 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.107851 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.108036 11832 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.108289 31926 leader_election.cc:302] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.108243 29324 catalog_manager.cc:5484] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:04.108568 8697 raft_consensus.cc:2781] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.108620 8697 raft_consensus.cc:482] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.108664 8697 raft_consensus.cc:3037] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.109639 8697 raft_consensus.cc:504] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.110333 8697 leader_election.cc:288] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.110987 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeab77cba6cf431e89141d4154c41a9b" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:01:04.111104 29863 raft_consensus.cc:3037] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.111299 11869 ts_tablet_manager.cc:1888] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:04.111382 11869 log.cc:1196] T ec8e4ac081604ec8a3f463f740d9119c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/ec8e4ac081604ec8a3f463f740d9119c
I20240502 14:01:04.112004 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet ec8e4ac081604ec8a3f463f740d9119c (table rebalancer_extra_table_6 [id=cf1fa19202d04c238e92c0b65e0162a7]) successfully deleted
I20240502 14:01:04.112031 29863 raft_consensus.cc:2454] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:04.112288 31926 leader_election.cc:302] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:01:04.112434 8697 raft_consensus.cc:2781] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.112519 8697 raft_consensus.cc:686] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:01:04.112639 8697 consensus_queue.cc:237] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
W20240502 14:01:04.113243 31925 leader_election.cc:334] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.115254 29324 catalog_manager.cc:5484] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:04.119534 10850 raft_consensus.cc:482] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.119638 10850 raft_consensus.cc:504] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.120043 10850 leader_election.cc:288] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:04.120615 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541806c11f074316a5342050427f4849" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
W20240502 14:01:04.121217 29535 leader_election.cc:334] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.121285 29535 leader_election.cc:302] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.121412 10850 raft_consensus.cc:2726] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.131424 10805 raft_consensus.cc:482] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.131537 10805 raft_consensus.cc:504] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.131924 10805 leader_election.cc:288] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:04.132447 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22543bedc3f043bba1d05b675a8f3c75" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:04.132572 31990 raft_consensus.cc:2454] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:04.133186 32188 leader_election.cc:302] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:04.133337 10805 raft_consensus.cc:2781] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.133407 10805 raft_consensus.cc:482] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.133461 10805 raft_consensus.cc:3037] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.133482 32188 leader_election.cc:334] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.134337 10805 raft_consensus.cc:504] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.134685 10805 leader_election.cc:288] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:04.135152 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22543bedc3f043bba1d05b675a8f3c75" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:04.135267 31990 raft_consensus.cc:3037] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.135705 32188 leader_election.cc:334] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.136210 31990 raft_consensus.cc:2454] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:04.136458 32188 leader_election.cc:302] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.136602 10805 raft_consensus.cc:2781] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.136703 10805 raft_consensus.cc:686] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:04.136852 10805 consensus_queue.cc:237] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.139811 29324 catalog_manager.cc:5484] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:04.149881 10012 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.149981 10012 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.150424 10012 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:04.150935 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:04.151058 31990 raft_consensus.cc:2454] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
I20240502 14:01:04.151425 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:01:04.151542 29602 raft_consensus.cc:2454] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 1.
W20240502 14:01:04.151834 31794 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.151991 31794 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.152318 10012 raft_consensus.cc:2781] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.152374 10012 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.152412 10012 raft_consensus.cc:3037] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.153525 10012 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.153890 10012 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:04.154268 8697 raft_consensus.cc:482] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.154351 8697 raft_consensus.cc:504] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.154780 8697 leader_election.cc:288] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:04.154863 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:04.154973 31990 raft_consensus.cc:3037] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.155320 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f77f3b6bf14a46b2e41c91125f6bd7" candidate_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:01:04.155414 29602 raft_consensus.cc:3037] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.156018 31990 raft_consensus.cc:2454] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 2.
I20240502 14:01:04.156247 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:04.156347 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:04.156390 29733 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:01:04.156397 29602 raft_consensus.cc:2454] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e01c4f581843e9ad4ced97f0072dc4 in term 2.
I20240502 14:01:04.156450 32253 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:01:04.156913 31927 leader_election.cc:302] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:04.157301 8697 raft_consensus.cc:2781] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.157353 8697 raft_consensus.cc:482] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.157389 8697 raft_consensus.cc:3037] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.157496 31925 leader_election.cc:334] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:04.158243 31794 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.158282 8697 raft_consensus.cc:504] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.158685 8697 leader_election.cc:288] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:04.159003 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:01:04.159124 32253 raft_consensus.cc:3037] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.159685 31925 leader_election.cc:334] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.159858 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5733940f27f345d8a314a9e51d2a455e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:04.159952 29733 raft_consensus.cc:3037] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.160197 32253 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:04.160830 29733 raft_consensus.cc:2454] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:04.161473 31926 leader_election.cc:302] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.162101 31794 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.162267 8697 raft_consensus.cc:2781] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.162326 10012 raft_consensus.cc:2781] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.162356 8697 raft_consensus.cc:686] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:01:04.162415 10012 raft_consensus.cc:686] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 LEADER]: Becoming Leader. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:01:04.162484 8697 consensus_queue.cc:237] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.162518 10012 consensus_queue.cc:237] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.164678 9471 raft_consensus.cc:482] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.164773 9471 raft_consensus.cc:504] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.165036 29324 catalog_manager.cc:5484] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:04.165186 9471 leader_election.cc:288] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:04.165534 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198b0a675f184a268bf714cf778408a4" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:04.165637 29733 raft_consensus.cc:2454] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:04.165537 29328 catalog_manager.cc:5484] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133). New cstate: current_term: 2 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
W20240502 14:01:04.167100 29797 leader_election.cc:334] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.167454 29798 leader_election.cc:302] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.167857 11058 raft_consensus.cc:2781] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.167923 11058 raft_consensus.cc:482] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.167973 11058 raft_consensus.cc:3037] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.168974 11058 raft_consensus.cc:504] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.169049 10850 raft_consensus.cc:482] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.169147 10850 raft_consensus.cc:504] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:04.169296 11058 leader_election.cc:288] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:04.169473 10850 leader_election.cc:288] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:04.170305 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198b0a675f184a268bf714cf778408a4" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:04.170423 29733 raft_consensus.cc:3037] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.170804 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50a38153ff44d1c988838bb39884f20" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:01:04.170933 31861 raft_consensus.cc:2454] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:01:04.171433 29733 raft_consensus.cc:2454] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
W20240502 14:01:04.171510 29797 leader_election.cc:334] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.171608 29535 leader_election.cc:302] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:04.171730 29798 leader_election.cc:302] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.171870 11058 raft_consensus.cc:2781] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.171911 10850 raft_consensus.cc:2781] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.171945 11058 raft_consensus.cc:686] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:04.171970 10850 raft_consensus.cc:482] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.172039 10850 raft_consensus.cc:3037] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.172055 11058 consensus_queue.cc:237] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.172822 10850 raft_consensus.cc:504] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:04.173142 10850 leader_election.cc:288] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:04.173481 29535 leader_election.cc:334] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.174311 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50a38153ff44d1c988838bb39884f20" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:01:04.174422 31861 raft_consensus.cc:3037] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.174551 29328 catalog_manager.cc:5484] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:04.175501 31861 raft_consensus.cc:2454] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 2.
I20240502 14:01:04.175675 10805 raft_consensus.cc:482] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.175766 10805 raft_consensus.cc:504] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.176199 10805 leader_election.cc:288] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:04.176708 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a34a1381cf84af6b560101d95447e6d" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:04.176847 29470 raft_consensus.cc:2454] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:04.177153 32190 leader_election.cc:302] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
W20240502 14:01:04.177245 32188 leader_election.cc:334] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.177393 10805 raft_consensus.cc:2781] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.177459 10805 raft_consensus.cc:482] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.177512 10805 raft_consensus.cc:3037] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.178525 10805 raft_consensus.cc:504] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.178902 10805 leader_election.cc:288] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
W20240502 14:01:04.179955 32188 leader_election.cc:334] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.180866 11779 raft_consensus.cc:482] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.180949 11779 raft_consensus.cc:504] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.181317 11779 leader_election.cc:288] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:04.183661 29535 leader_election.cc:302] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:04.184502 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63de5b3f35e4486859c9361925689d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" is_pre_election: true
I20240502 14:01:04.184612 31859 raft_consensus.cc:2454] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 1.
I20240502 14:01:04.184590 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a34a1381cf84af6b560101d95447e6d" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:04.184691 29470 raft_consensus.cc:3037] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.185002 11779 raft_consensus.cc:2781] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.185073 11779 raft_consensus.cc:686] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [term 2 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:01:04.185192 11779 consensus_queue.cc:237] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:04.185815 29470 raft_consensus.cc:2454] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:04.186224 32190 leader_election.cc:302] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:04.186872 29535 leader_election.cc:334] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.187151 10013 raft_consensus.cc:2781] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.187323 10013 raft_consensus.cc:686] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:04.187443 10013 consensus_queue.cc:237] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.188426 29535 leader_election.cc:302] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
W20240502 14:01:04.189020 29535 leader_election.cc:334] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.189213 11779 raft_consensus.cc:2781] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.189260 11779 raft_consensus.cc:482] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.189302 11779 raft_consensus.cc:3037] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.190320 11779 raft_consensus.cc:504] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.190627 11779 leader_election.cc:288] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:04.191967 29322 catalog_manager.cc:5484] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 2 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:04.192099 29323 catalog_manager.cc:5484] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:01:04.195593 29535 leader_election.cc:334] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.195804 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63de5b3f35e4486859c9361925689d9" candidate_uuid: "d6e2e2c10def4331bec17bd110c9b423" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:01:04.195916 31860 raft_consensus.cc:3037] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.196954 31860 raft_consensus.cc:2454] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e2e2c10def4331bec17bd110c9b423 in term 2.
I20240502 14:01:04.197201 9471 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.197295 9471 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.197587 29535 leader_election.cc:302] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.197710 9471 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:04.197913 10850 raft_consensus.cc:2781] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.197997 10850 raft_consensus.cc:686] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [term 2 LEADER]: Becoming Leader. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Running, Role: LEADER
I20240502 14:01:04.198120 10850 consensus_queue.cc:237] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.198985 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:01:04.200075 29797 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.200170 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 8973f2527acb48f5ad36589652de73e1, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.200314 9471 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:04.200906 29799 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
I20240502 14:01:04.203408 29324 catalog_manager.cc:5484] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130). New cstate: current_term: 2 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:04.203953 11477 raft_consensus.cc:482] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.204049 11477 raft_consensus.cc:504] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.204456 11477 leader_election.cc:288] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.205195 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65229e977b8543639fb27bbbc3b5e797" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:04.205320 29733 raft_consensus.cc:2454] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:01:04.205531 11832 raft_consensus.cc:482] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.205649 11832 raft_consensus.cc:504] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.205718 31926 leader_election.cc:302] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:04.205899 11477 raft_consensus.cc:2781] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.205957 11477 raft_consensus.cc:482] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.206009 11477 raft_consensus.cc:3037] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.206122 11832 leader_election.cc:288] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:04.206964 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541806c11f074316a5342050427f4849" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423" is_pre_election: true
I20240502 14:01:04.207088 29602 raft_consensus.cc:2454] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:01:04.207201 11477 raft_consensus.cc:504] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
W20240502 14:01:04.207319 29404 leader_election.cc:334] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.207464 29404 leader_election.cc:302] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.207516 11477 leader_election.cc:288] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.207659 11832 raft_consensus.cc:2781] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.207710 11832 raft_consensus.cc:482] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.207753 11832 raft_consensus.cc:3037] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.207839 31925 leader_election.cc:334] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.208405 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65229e977b8543639fb27bbbc3b5e797" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:04.208508 29733 raft_consensus.cc:3037] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.208837 11832 raft_consensus.cc:504] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.209185 11832 leader_election.cc:288] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:04.209630 29733 raft_consensus.cc:2454] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:04.210158 31926 leader_election.cc:302] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
W20240502 14:01:04.210251 31925 leader_election.cc:334] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.210337 11477 raft_consensus.cc:2781] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.210422 11477 raft_consensus.cc:686] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:01:04.210563 11477 consensus_queue.cc:237] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.211460 29602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541806c11f074316a5342050427f4849" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d6e2e2c10def4331bec17bd110c9b423"
I20240502 14:01:04.211609 29602 raft_consensus.cc:3037] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.212632 29404 leader_election.cc:334] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.212719 29602 raft_consensus.cc:2454] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 2.
I20240502 14:01:04.212936 10805 raft_consensus.cc:482] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.213027 29404 leader_election.cc:302] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.213057 10805 raft_consensus.cc:504] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.213227 11832 raft_consensus.cc:2781] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.213320 11832 raft_consensus.cc:686] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:01:04.213446 10805 leader_election.cc:288] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.213443 11832 consensus_queue.cc:237] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:04.213817 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c8871730c4e2ba9f1320381159331" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:04.213943 29864 raft_consensus.cc:2454] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:04.214118 29324 catalog_manager.cc:5484] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:04.214520 32189 leader_election.cc:302] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:04.214720 10805 raft_consensus.cc:2781] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.214763 10805 raft_consensus.cc:482] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.214799 10805 raft_consensus.cc:3037] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:04.214931 32188 leader_election.cc:334] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.216035 9471 raft_consensus.cc:482] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.216130 9471 raft_consensus.cc:504] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:04.216176 10805 raft_consensus.cc:504] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.216470 9471 leader_election.cc:288] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:04.216882 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cda992fe784d9f9015e6a3d431f1cd" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:04.216868 29322 catalog_manager.cc:5484] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 2 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:04.216990 32253 raft_consensus.cc:2454] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:04.217207 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c8871730c4e2ba9f1320381159331" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:01:04.217312 29864 raft_consensus.cc:3037] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.217475 29799 leader_election.cc:302] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:04.217689 9471 raft_consensus.cc:2781] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.217737 9471 raft_consensus.cc:482] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.217773 9471 raft_consensus.cc:3037] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.218418 29864 raft_consensus.cc:2454] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:04.218756 9471 raft_consensus.cc:504] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:04.219131 9471 leader_election.cc:288] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:04.219305 32189 leader_election.cc:302] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:04.219555 10013 raft_consensus.cc:2781] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.219624 10013 raft_consensus.cc:686] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:04.219686 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cda992fe784d9f9015e6a3d431f1cd" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:01:04.219727 10013 consensus_queue.cc:237] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.219823 32253 raft_consensus.cc:3037] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.220835 32253 raft_consensus.cc:2454] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
W20240502 14:01:04.221518 29797 leader_election.cc:334] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.221560 29799 leader_election.cc:302] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
W20240502 14:01:04.221633 29797 leader_election.cc:334] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:04.222666 32188 leader_election.cc:334] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.222817 10013 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.222945 10013 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:04.222939 29324 catalog_manager.cc:5484] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:04.223297 10013 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:04.223888 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:04.224004 29864 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:04.224820 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
I20240502 14:01:04.224922 11058 raft_consensus.cc:2781] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.225069 11058 raft_consensus.cc:686] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:04.225196 11058 consensus_queue.cc:237] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:04.225780 11058 raft_consensus.cc:482] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.225857 11058 raft_consensus.cc:504] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.226193 11058 leader_election.cc:288] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.227111 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b415bd7b51fc4e4a8c6fa6083d989bfe" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
W20240502 14:01:04.227173 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.227227 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.227252 29732 raft_consensus.cc:2454] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:04.227684 29798 leader_election.cc:302] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
W20240502 14:01:04.227978 29797 leader_election.cc:334] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.228142 10013 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:04.228926 9471 raft_consensus.cc:2781] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.228991 9471 raft_consensus.cc:482] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.229056 9471 raft_consensus.cc:3037] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.229171 29324 catalog_manager.cc:5484] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:04.230024 9471 raft_consensus.cc:504] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.230369 9471 leader_election.cc:288] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.230804 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b415bd7b51fc4e4a8c6fa6083d989bfe" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:04.230943 29732 raft_consensus.cc:3037] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.231133 11846 raft_consensus.cc:482] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.231199 11846 raft_consensus.cc:504] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.231542 11846 leader_election.cc:288] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:04.231946 29797 leader_election.cc:334] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.232002 29732 raft_consensus.cc:2454] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:01:04.232223 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c08999e4e7f4d0bb6a03199ee676f76" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:04.232257 29798 leader_election.cc:302] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.232367 29470 raft_consensus.cc:2454] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:01:04.232601 9471 raft_consensus.cc:2781] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 14:01:04.232599 29666 leader_election.cc:334] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.232703 29668 leader_election.cc:302] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:04.232748 9471 raft_consensus.cc:686] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:04.232965 11846 raft_consensus.cc:2781] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:04.233024 11846 raft_consensus.cc:482] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:04.233081 11846 raft_consensus.cc:3037] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.232954 9471 consensus_queue.cc:237] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:04.234066 11846 raft_consensus.cc:504] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.234403 11846 leader_election.cc:288] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.234794 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c08999e4e7f4d0bb6a03199ee676f76" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:04.234930 29470 raft_consensus.cc:3037] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.235919 29470 raft_consensus.cc:2454] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
I20240502 14:01:04.236374 29668 leader_election.cc:302] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:04.236594 11846 raft_consensus.cc:2781] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.236670 11846 raft_consensus.cc:686] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:01:04.236783 11846 consensus_queue.cc:237] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:04.238430 10805 leader_election.cc:288] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:04.241737 29320 catalog_manager.cc:5484] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 2 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:01:04.242580 29666 leader_election.cc:334] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.247797 29329 catalog_manager.cc:5484] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
W20240502 14:01:04.248195 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.251205 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.301131 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:04.302137 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.303743 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:04.308966 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:04.314510 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:04.315457 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:04.317560 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:04.328536 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:04.334808 31860 raft_consensus.cc:1261] T a8e5364d50da4cfb8fbbf21913324bc6 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.335189 11917 consensus_queue.cc:1035] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:04.350489 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.355940 29864 raft_consensus.cc:1261] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:04.356381 11775 consensus_queue.cc:1035] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:04.356557 29729 raft_consensus.cc:1261] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.356935 11917 consensus_queue.cc:1035] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:04.366915 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.396281 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.406737 29602 raft_consensus.cc:1261] T 2868541b54c447219db948c92b9a132b P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.407181 10013 consensus_queue.cc:1035] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:04.417563 31990 consensus_queue.cc:237] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:04.418686 29864 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:04.418758 29729 raft_consensus.cc:1261] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:04.419065 8697 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:04.419281 11477 consensus_queue.cc:1035] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:04.421285 8697 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:04.423003 29729 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:04.423830 29864 raft_consensus.cc:2932] T a1116d9f165b488796ab466f9a866471 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:04.424746 29327 catalog_manager.cc:5484] T a1116d9f165b488796ab466f9a866471 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index 5 to 6, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:04.429055 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet a1116d9f165b488796ab466f9a866471 with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47700
I20240502 14:01:04.429446 11989 tablet_replica.cc:330] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:01:04.429600 11989 raft_consensus.cc:2229] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:04.429759 11989 raft_consensus.cc:2258] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:04.430294 11989 ts_tablet_manager.cc:1875] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:04.433028 11989 ts_tablet_manager.cc:1888] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:01:04.433159 11989 log.cc:1196] T a1116d9f165b488796ab466f9a866471 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/a1116d9f165b488796ab466f9a866471
I20240502 14:01:04.433998 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet a1116d9f165b488796ab466f9a866471 (table rebalancer_extra_table_4 [id=b89f4a5a7266450baa63e6dbdf38e5d4]) successfully deleted
W20240502 14:01:04.435221 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.443706 29469 raft_consensus.cc:1261] T f8b939db0e5b464dbcecbf8e18cccadf P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:04.444113 11444 consensus_queue.cc:1035] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:04.488905 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.530233 31990 raft_consensus.cc:1261] T 6df051496f954569983892bfe828f653 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.530664 10013 consensus_queue.cc:1035] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:04.534132 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.558468 31990 raft_consensus.cc:1261] T 22543bedc3f043bba1d05b675a8f3c75 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.558971 10805 consensus_queue.cc:1035] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:04.559649 29729 raft_consensus.cc:1261] T 5733940f27f345d8a314a9e51d2a455e P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:04.560992 8697 consensus_queue.cc:1035] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:01:04.565366 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.567778 31990 raft_consensus.cc:1261] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:04.568898 11705 consensus_queue.cc:1035] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:04.589736 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:04.590155 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:04.592512 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:04.593021 29469 raft_consensus.cc:1261] T 70367c7c718d4d9195f0af7e69bdf2ec P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:04.594215 10013 consensus_queue.cc:1035] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:04.595579 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:04.597018 29602 raft_consensus.cc:1261] T 09f77f3b6bf14a46b2e41c91125f6bd7 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:04.597487 10012 consensus_queue.cc:1035] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:04.597904 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:04.599053 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:04.599157 29600 raft_consensus.cc:1261] T 541806c11f074316a5342050427f4849 P d6e2e2c10def4331bec17bd110c9b423 [term 2 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:04.599567 11832 consensus_queue.cc:1035] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:04.617178 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.620203 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.622076 29470 raft_consensus.cc:1261] T 3a34a1381cf84af6b560101d95447e6d P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:04.622710 10805 consensus_queue.cc:1035] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:04.622924 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.626688 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.638898 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:04.648209 31860 raft_consensus.cc:1261] T 6d3a84c1d004412db381854147330dfc P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:04.651134 10850 consensus_queue.cc:1035] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:04.653986 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.654023 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.658296 29729 raft_consensus.cc:1261] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.658809 8697 consensus_queue.cc:1035] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:04.662746 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.663372 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.668336 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.689704 32253 raft_consensus.cc:1261] T 82cda992fe784d9f9015e6a3d431f1cd P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:04.690212 9471 consensus_queue.cc:1035] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:04.697083 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.700196 29863 raft_consensus.cc:1261] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.700874 11984 consensus_queue.cc:1035] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:04.701651 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.706504 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.707455 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:04.725797 32253 raft_consensus.cc:1261] T 5733940f27f345d8a314a9e51d2a455e P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:04.726270 8697 consensus_queue.cc:1035] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:04.729022 31860 raft_consensus.cc:1261] T a50a38153ff44d1c988838bb39884f20 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.729619 11779 consensus_queue.cc:1035] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:04.737279 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:04.750983 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:04.757645 31860 raft_consensus.cc:1261] T a63de5b3f35e4486859c9361925689d9 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.758096 10850 consensus_queue.cc:1035] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:04.768013 29732 raft_consensus.cc:1261] T 198b0a675f184a268bf714cf778408a4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.768491 9471 consensus_queue.cc:1035] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:04.793675 29863 raft_consensus.cc:1261] T ed0c8871730c4e2ba9f1320381159331 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.794207 10805 consensus_queue.cc:1035] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:04.805589 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:04.829336 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:04.834096 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:04.838869 29469 raft_consensus.cc:1261] T 4c08999e4e7f4d0bb6a03199ee676f76 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.839288 11575 consensus_queue.cc:1035] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:04.843796 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:04.845089 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:04.847473 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:04.850680 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:04.851328 29732 raft_consensus.cc:1261] T b415bd7b51fc4e4a8c6fa6083d989bfe P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:04.851701 11058 consensus_queue.cc:1035] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:04.868938 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:04.923152 29732 consensus_queue.cc:237] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:04.924285 32253 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:04.924357 31991 raft_consensus.cc:1261] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:04.924615 11444 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:04.924768 11575 consensus_queue.cc:1035] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:04.926577 11444 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:04.927088 31991 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:04.927534 32253 raft_consensus.cc:2932] T 92cc4394003840a5b1d59b0fe72c5164 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:04.931356 29329 catalog_manager.cc:5484] T 92cc4394003840a5b1d59b0fe72c5164 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 6 to 7, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:04.935302 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet 92cc4394003840a5b1d59b0fe72c5164 with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57986
I20240502 14:01:04.936185 12142 tablet_replica.cc:330] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:01:04.936430 12142 raft_consensus.cc:2229] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:04.936723 12142 raft_consensus.cc:2258] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:04.937304 12142 ts_tablet_manager.cc:1875] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:04.939882 12142 ts_tablet_manager.cc:1888] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:04.940004 12142 log.cc:1196] T 92cc4394003840a5b1d59b0fe72c5164 P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/92cc4394003840a5b1d59b0fe72c5164
I20240502 14:01:04.940579 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet 92cc4394003840a5b1d59b0fe72c5164 (table rebalance_test_table_11 [id=1f4578a2cedc446faadf398b702e42b3]) successfully deleted
W20240502 14:01:04.984136 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:05.006491 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.052526 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.079392 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:05.085254 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:05.088949 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:05.094712 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:05.099042 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:05.105813 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:05.123471 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.132891 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:05.164644 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.166425 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.187078 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.188936 29600 consensus_queue.cc:237] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:05.190124 29732 raft_consensus.cc:1261] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:05.190173 29864 raft_consensus.cc:1261] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:05.190538 11775 consensus_queue.cc:1035] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:05.190711 10850 consensus_queue.cc:1035] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:05.193416 10850 raft_consensus.cc:2932] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:01:05.193636 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:05.193887 29732 raft_consensus.cc:2932] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:05.195241 29864 raft_consensus.cc:2932] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:05.197633 29328 catalog_manager.cc:5484] T 4e69badccfbc416f92d982cf394ea0cd P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:05.200801 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69badccfbc416f92d982cf394ea0cd with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:01:05.201575 12212 tablet_replica.cc:330] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:01:05.201746 12212 raft_consensus.cc:2229] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:05.201869 12212 raft_consensus.cc:2258] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:05.202304 12212 ts_tablet_manager.cc:1875] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:05.205027 12212 ts_tablet_manager.cc:1888] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:05.205107 12212 log.cc:1196] T 4e69badccfbc416f92d982cf394ea0cd P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/4e69badccfbc416f92d982cf394ea0cd
I20240502 14:01:05.205590 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 4e69badccfbc416f92d982cf394ea0cd (table rebalancer_extra_table_54 [id=0abe1feaf8a647ae894c60970f5eff90]) successfully deleted
W20240502 14:01:05.222756 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.241395 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.245572 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.268396 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.280156 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.281313 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.328059 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:05.335278 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:05.343291 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.344274 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.345329 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:05.347033 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:05.349051 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.376878 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:05.377132 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:05.378543 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.396276 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.396435 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.399794 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.400666 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.402036 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:05.405287 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.432282 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.469933 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:05.548794 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.586215 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:05.588665 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:05.599067 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.601184 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:05.601783 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.602458 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:05.604647 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:05.609401 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.611685 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:05.616413 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:05.618034 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.626502 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.633440 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.666872 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.690678 29602 consensus_queue.cc:237] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:05.692301 31990 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:05.692416 29864 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:01:05.692652 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:05.692883 29535 consensus_peers.cc:485] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d2ab1ded2c241038d5685678aa819a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:05.692958 10850 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:05.693746 10850 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:05.695225 11779 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:05.695616 29864 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:05.698055 31990 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:05.699047 29320 catalog_manager.cc:5484] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
W20240502 14:01:05.706378 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:05.727658 29864 consensus_queue.cc:237] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:05.729111 32255 raft_consensus.cc:1261] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:05.729280 29602 raft_consensus.cc:1261] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:05.729745 11058 consensus_queue.cc:1035] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:05.729918 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:05.730079 11058 consensus_queue.cc:1035] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:05.736200 32255 raft_consensus.cc:2932] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
W20240502 14:01:05.736949 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.735986 12378 raft_consensus.cc:2932] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:05.739578 29326 catalog_manager.cc:5484] T bf3e82b3dc394648a13eafe4ec832fb0 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:05.742242 29602 raft_consensus.cc:2932] T bf3e82b3dc394648a13eafe4ec832fb0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } }
I20240502 14:01:05.760969 29729 consensus_queue.cc:237] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:05.762486 31990 raft_consensus.cc:1261] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:05.762589 29602 raft_consensus.cc:1261] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:05.762941 11846 consensus_queue.cc:1035] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:05.763150 29668 consensus_peers.cc:485] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90c275c029644bb5be541811dc185f92. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:05.763234 12397 consensus_queue.cc:1035] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:05.764746 12397 raft_consensus.cc:2932] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:05.765177 31990 raft_consensus.cc:2932] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:05.767695 29602 raft_consensus.cc:2932] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:05.769328 29321 catalog_manager.cc:5484] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:05.788579 12378 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:05.788705 12378 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:05.789198 12378 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:05.789500 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:05.789630 32255 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
W20240502 14:01:05.790053 29799 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:05.790510 29797 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:05.790580 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:05.790724 12378 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:05.791527 29729 consensus_queue.cc:237] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:05.792928 31860 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:05.792949 29469 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:01:05.793165 29666 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fc14b4bd42841df91972e0c5d02faa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:05.793320 11846 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:05.793481 11575 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:05.795621 11575 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:05.796048 31860 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:05.797528 29469 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:05.800022 29326 catalog_manager.cc:5484] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 5 to 6, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:05.818981 29864 consensus_queue.cc:237] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:05.820268 29602 raft_consensus.cc:1261] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:01:05.820576 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7650009dd8954f11ab2ecafef765692c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:05.820739 11058 consensus_queue.cc:1035] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:01:05.821374 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:05.913833 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:05.915701 12322 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:05.915822 12322 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:05.916307 12322 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:05.920939 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:05.921288 32255 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:05.921769 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:05.921854 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:05.921942 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:05.922297 12322 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:05.954427 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:06.178906 12437 ts_tablet_manager.cc:902] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:06.179349 12437 tablet_copy_client.cc:308] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:01:06.179769 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet 8d2ab1ded2c241038d5685678aa819a8 from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:49783)
I20240502 14:01:06.179880 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet 8d2ab1ded2c241038d5685678aa819a8 from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:49783: session id = f0e01c4f581843e9ad4ced97f0072dc4-8d2ab1ded2c241038d5685678aa819a8
I20240502 14:01:06.180650 29622 tablet_copy_source_session.cc:205] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:06.181429 12437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2ab1ded2c241038d5685678aa819a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:06.183317 12437 tablet_copy_client.cc:783] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:06.183472 12437 tablet_copy_client.cc:647] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:06.184654 12437 tablet_copy_client.cc:523] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:06.185854 12437 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:01:06.205329 12437 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:06.205603 12437 tablet_bootstrap.cc:492] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:01:06.205790 12437 ts_tablet_manager.cc:1367] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240502 14:01:06.206211 12437 raft_consensus.cc:348] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:06.206319 12437 raft_consensus.cc:729] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:01:06.206434 12437 consensus_queue.cc:260] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:06.206702 12437 ts_tablet_manager.cc:1398] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:06.207172 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-8d2ab1ded2c241038d5685678aa819a8 received from {username='slave'} at 127.22.55.133:49783
I20240502 14:01:06.207257 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-8d2ab1ded2c241038d5685678aa819a8 on tablet 8d2ab1ded2c241038d5685678aa819a8 with peer f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:06.306744 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:06.311805 12443 ts_tablet_manager.cc:902] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:06.312235 12443 tablet_copy_client.cc:308] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:01:06.312628 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 90c275c029644bb5be541811dc185f92 from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:38447)
I20240502 14:01:06.312736 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 90c275c029644bb5be541811dc185f92 from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:38447: session id = 1d00d3a47c52478894cf8c66959ef7bd-90c275c029644bb5be541811dc185f92
I20240502 14:01:06.313582 29753 tablet_copy_source_session.cc:205] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:06.314378 12443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c275c029644bb5be541811dc185f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:06.316895 12443 tablet_copy_client.cc:783] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:06.317123 12443 tablet_copy_client.cc:647] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:06.318044 12443 tablet_copy_client.cc:523] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:06.319185 12443 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:01:06.323124 12443 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:06.323367 12443 tablet_bootstrap.cc:492] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:01:06.323514 12443 ts_tablet_manager.cc:1367] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:01:06.323871 12443 raft_consensus.cc:348] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:06.323968 12443 raft_consensus.cc:729] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:01:06.324203 12443 consensus_queue.cc:260] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:06.325356 12443 ts_tablet_manager.cc:1398] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:06.325824 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-90c275c029644bb5be541811dc185f92 received from {username='slave'} at 127.22.55.129:38447
I20240502 14:01:06.325906 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-90c275c029644bb5be541811dc185f92 on tablet 90c275c029644bb5be541811dc185f92 with peer 1d00d3a47c52478894cf8c66959ef7bd
W20240502 14:01:06.336807 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:06.401429 12449 ts_tablet_manager.cc:902] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:06.401883 12449 tablet_copy_client.cc:308] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:01:06.402366 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 1fc14b4bd42841df91972e0c5d02faa4 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:50019)
I20240502 14:01:06.402446 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 1fc14b4bd42841df91972e0c5d02faa4 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:50019: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-1fc14b4bd42841df91972e0c5d02faa4
I20240502 14:01:06.403290 29753 tablet_copy_source_session.cc:205] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:06.404130 12449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc14b4bd42841df91972e0c5d02faa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:06.406176 12449 tablet_copy_client.cc:783] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:06.406337 12449 tablet_copy_client.cc:647] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:06.407368 12449 tablet_copy_client.cc:523] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:06.408458 12449 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:01:06.423102 12437 ts_tablet_manager.cc:902] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:06.423566 12437 tablet_copy_client.cc:308] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:01:06.423966 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet 7650009dd8954f11ab2ecafef765692c from peer f0e01c4f581843e9ad4ced97f0072dc4 ({username='slave'} at 127.22.55.133:44851)
I20240502 14:01:06.424079 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet 7650009dd8954f11ab2ecafef765692c from peer f0e01c4f581843e9ad4ced97f0072dc4 at {username='slave'} at 127.22.55.133:44851: session id = f0e01c4f581843e9ad4ced97f0072dc4-7650009dd8954f11ab2ecafef765692c
I20240502 14:01:06.425037 29884 tablet_copy_source_session.cc:205] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:06.425835 12437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7650009dd8954f11ab2ecafef765692c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:06.427912 12437 tablet_copy_client.cc:783] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:06.428058 12437 tablet_copy_client.cc:647] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:06.429152 12437 tablet_copy_client.cc:523] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:06.429852 12449 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:06.430079 12449 tablet_bootstrap.cc:492] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:01:06.430225 12449 ts_tablet_manager.cc:1367] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.005s
I20240502 14:01:06.430289 12437 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap starting.
I20240502 14:01:06.430589 12449 raft_consensus.cc:348] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:06.430709 12449 raft_consensus.cc:729] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:01:06.430837 12449 consensus_queue.cc:260] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:06.431241 12449 ts_tablet_manager.cc:1398] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 14:01:06.431874 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-1fc14b4bd42841df91972e0c5d02faa4 received from {username='slave'} at 127.22.55.134:50019
I20240502 14:01:06.431934 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-1fc14b4bd42841df91972e0c5d02faa4 on tablet 1fc14b4bd42841df91972e0c5d02faa4 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:01:06.454670 12437 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:06.455068 12437 tablet_bootstrap.cc:492] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: Bootstrap complete.
I20240502 14:01:06.455246 12437 ts_tablet_manager.cc:1367] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240502 14:01:06.455677 12437 raft_consensus.cc:348] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:06.455909 12437 raft_consensus.cc:729] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Initialized, Role: FOLLOWER
I20240502 14:01:06.456022 12437 consensus_queue.cc:260] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:06.456398 12437 ts_tablet_manager.cc:1398] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:06.456918 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-7650009dd8954f11ab2ecafef765692c received from {username='slave'} at 127.22.55.133:44851
I20240502 14:01:06.457007 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session f0e01c4f581843e9ad4ced97f0072dc4-7650009dd8954f11ab2ecafef765692c on tablet 7650009dd8954f11ab2ecafef765692c with peer f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:06.461068 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:06.698170 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:06.706751 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:06.747047 29470 raft_consensus.cc:1203] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:01:06.760932 31860 raft_consensus.cc:1203] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240502 14:01:06.775611 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:06.844403 31990 raft_consensus.cc:1203] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240502 14:01:06.891525 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:06.940336 31860 raft_consensus.cc:1203] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:06.941812 11058 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:06.942180 31861 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:06.943107 11058 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35688 (request call id 0) took 1124 ms. Trace:
I20240502 14:01:06.943190 11058 rpcz_store.cc:276] 0502 14:01:05.818525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:01:05.818579 (+ 54us) service_pool.cc:225] Handling call
0502 14:01:05.819557 (+ 978us) log.cc:842] Serialized 489 byte log entry
0502 14:01:06.943092 (+1123535us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":44,"raft.queue_time_us":111,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":335,"spinlock_wait_cycles":82304,"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":116,"wal-append.run_cpu_time_us":257,"wal-append.run_wall_time_us":1000320}
W20240502 14:01:06.948875 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:06.949672 29329 catalog_manager.cc:5484] T 7650009dd8954f11ab2ecafef765692c P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:06.955410 29600 raft_consensus.cc:2932] T 7650009dd8954f11ab2ecafef765692c P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
W20240502 14:01:07.032606 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.043171 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.107110 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.108888 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:07.115486 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.115581 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:07.119040 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:07.119906 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.121803 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:07.125564 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.139554 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:07.143841 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:07.146916 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.151672 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.152354 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.156071 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:07.163328 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:07.173993 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.174086 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.264827 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.269063 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.271870 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.312958 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:07.318678 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.322746 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:07.325248 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.325935 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:07.328883 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:07.335052 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:07.339458 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.341542 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:07.362170 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:07.368157 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.431078 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:07.436344 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.479393 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.507308 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.520026 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:07.522943 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.547564 29713 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:07.547726 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:07.551084 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:07.551088 31970 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:07.554497 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.561878 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:07.562830 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:07.565696 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:07.575404 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:07.617121 29732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4"
dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44696
I20240502 14:01:07.617270 29732 raft_consensus.cc:595] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Received request to transfer leadership
W20240502 14:01:07.625389 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.646459 10805 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:07.646571 10805 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:07.647061 10805 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:07.647456 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:07.647612 29469 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:07.648007 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:07.648377 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:07.648448 32188 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:07.648573 10805 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:07.674643 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.685309 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.696662 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.704288 11575 raft_consensus.cc:979] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: : Instructing follower 269b366a09ff435ea8ec8dc01d0f9f6e to start an election
I20240502 14:01:07.704397 11575 raft_consensus.cc:1067] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Signalling peer 269b366a09ff435ea8ec8dc01d0f9f6e to start an election
I20240502 14:01:07.705483 31990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4"
dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
from {username='slave'} at 127.22.55.131:46293
I20240502 14:01:07.705632 31990 raft_consensus.cc:482] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:07.705691 31990 raft_consensus.cc:3037] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:07.707274 31990 raft_consensus.cc:504] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:07.707715 31990 leader_election.cc:288] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:07.709659 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:07.709728 31860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:01:07.709782 29469 raft_consensus.cc:3037] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:07.709857 31860 raft_consensus.cc:3037] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:07.710824 31860 raft_consensus.cc:2454] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:07.710824 29469 raft_consensus.cc:2454] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:07.711408 31927 leader_election.cc:302] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:07.711587 12461 raft_consensus.cc:2781] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:07.711666 12461 raft_consensus.cc:686] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:01:07.711794 12461 consensus_queue.cc:237] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:07.713274 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc14b4bd42841df91972e0c5d02faa4" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:07.713400 29729 raft_consensus.cc:3032] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:07.713443 29729 raft_consensus.cc:729] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:01:07.713567 29729 consensus_queue.cc:260] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:07.713781 29729 raft_consensus.cc:3037] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:07.715063 29729 raft_consensus.cc:2454] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
W20240502 14:01:07.715997 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.716902 29324 catalog_manager.cc:5484] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
W20240502 14:01:07.737493 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.757222 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:07.759850 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:07.766067 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:07.773159 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:07.775640 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:07.780185 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:07.806474 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:07.815717 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.846180 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.892004 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.927242 11058 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:07.927368 11058 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:07.927974 11058 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:07.928841 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:07.928985 32255 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
W20240502 14:01:07.929507 29799 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:07.930022 29797 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:07.930086 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:07.930310 11058 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:07.937582 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:07.937625 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:07.951274 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:07.953017 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:07.956452 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.956614 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:07.966295 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.968214 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:07.971938 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.972038 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:07.976819 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:07.981065 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:07.995025 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:07.995204 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:07.996404 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.000834 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.000864 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:08.023403 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.033159 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.033784 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.035136 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.068284 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.099756 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 169 similar messages]
I20240502 14:01:08.101832 29729 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:08.102253 12590 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:01:08.116878 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:01:08.123558 31794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:01:08.135639 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 14:01:08.136355 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:08.148483 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.157639 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.159119 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 45 similar messages]
I20240502 14:01:08.161407 31861 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:08.162708 12590 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:08.169629 29470 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:08.170244 12784 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:01:08.173882 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:01:08.200541 29535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 97 similar messages]
I20240502 14:01:08.213932 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:08.215667 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:08.220283 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:08.220861 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:08.223937 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:08.225975 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:08.257954 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:08.288326 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:08.302094 29602 consensus_queue.cc:237] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:08.303231 31861 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:08.303279 31990 raft_consensus.cc:1261] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:08.303601 12566 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:08.303748 12358 consensus_queue.cc:1035] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:08.305516 12820 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:08.305874 31861 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:08.305932 31990 raft_consensus.cc:2932] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:08.310454 29327 catalog_manager.cc:5484] T 8d2ab1ded2c241038d5685678aa819a8 P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index 6 to 7, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:08.314217 29843 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2ab1ded2c241038d5685678aa819a8 with delete_type TABLET_DATA_TOMBSTONED (TS 4aebe6d7a3f34bf7b2bbbfbef1077463 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37732
I20240502 14:01:08.314543 12824 tablet_replica.cc:330] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: stopping tablet replica
I20240502 14:01:08.314673 12824 raft_consensus.cc:2229] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:08.315083 12824 raft_consensus.cc:2258] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:08.315632 12824 ts_tablet_manager.cc:1875] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:01:08.316072 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:08.318121 12824 ts_tablet_manager.cc:1888] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:08.318240 12824 log.cc:1196] T 8d2ab1ded2c241038d5685678aa819a8 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-3/wal/wals/8d2ab1ded2c241038d5685678aa819a8
I20240502 14:01:08.318921 29314 catalog_manager.cc:4830] TS 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): tablet 8d2ab1ded2c241038d5685678aa819a8 (table rebalance_test_table_1 [id=c537595292754404a0dc663dabe96b0d]) successfully deleted
W20240502 14:01:08.386783 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:08.407831 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.409955 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:08.445593 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:08.450345 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:08.454155 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:08.455049 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:08.458957 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:08.466951 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:08.478994 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:08.485152 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:08.661681 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:08.663728 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:08.665256 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:08.666710 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:08.675984 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:08.678988 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:08.698997 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:08.827751 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:08.870507 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:08.877372 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:08.879317 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:08.879535 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:08.886791 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:08.896689 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:08.897446 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:08.911201 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:08.923796 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:08.937660 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:09.113749 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:09.114524 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:09.117995 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:09.119419 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:09.125547 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:09.125800 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:09.132038 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:09.141655 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:09.351280 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:09.357103 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:09.374984 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:09.375270 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:09.378119 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:09.378646 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:09.379077 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:09.389132 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:09.408525 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:09.431891 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.457785 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.458912 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.461328 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.464547 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.521577 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:09.546172 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:09.585958 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:09.586763 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:09.593092 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:09.596259 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:09.598049 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:09.599275 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.614574 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:09.620000 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:09.627411 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:09.655747 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.657474 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.661334 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:09.669734 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.679567 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:09.683275 29729 consensus_queue.cc:237] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
W20240502 14:01:09.683368 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:09.684588 29470 raft_consensus.cc:1261] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:09.684666 31990 raft_consensus.cc:1261] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:09.685016 12800 consensus_queue.cc:1035] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:09.685178 11575 consensus_queue.cc:1035] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:09.686980 12800 raft_consensus.cc:2932] T 90c275c029644bb5be541811dc185f92 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:09.687372 29470 raft_consensus.cc:2932] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:09.687918 31990 raft_consensus.cc:2932] T 90c275c029644bb5be541811dc185f92 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:09.690791 29320 catalog_manager.cc:5484] T 90c275c029644bb5be541811dc185f92 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 2 to 3, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:09.697942 29582 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c275c029644bb5be541811dc185f92 with delete_type TABLET_DATA_TOMBSTONED (TS d6e2e2c10def4331bec17bd110c9b423 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55288
I20240502 14:01:09.707067 13184 tablet_replica.cc:330] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: stopping tablet replica
I20240502 14:01:09.707221 13184 raft_consensus.cc:2229] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:09.707357 13184 raft_consensus.cc:2258] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:09.707890 13184 ts_tablet_manager.cc:1875] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:09.710474 13184 ts_tablet_manager.cc:1888] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:09.710611 13184 log.cc:1196] T 90c275c029644bb5be541811dc185f92 P d6e2e2c10def4331bec17bd110c9b423: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-1/wal/wals/90c275c029644bb5be541811dc185f92
I20240502 14:01:09.711567 29313 catalog_manager.cc:4830] TS d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): tablet 90c275c029644bb5be541811dc185f92 (table rebalancer_extra_table_48 [id=3dcb94027f824c4ab69375cf1ae201b5]) successfully deleted
W20240502 14:01:09.727136 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.755875 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.782527 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.817592 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:09.823598 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:09.834976 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:09.838745 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:09.840314 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:09.847333 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:09.856654 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:09.867350 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:09.873520 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:09.881467 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:09.892633 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:09.945974 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:09.955400 10805 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:09.955518 10805 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:09.955976 10805 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:09.956285 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:09.956411 29863 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:09.957135 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:09.958117 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:09.958204 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:09.958384 10805 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:09.963481 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:09.995276 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:10.014051 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.052112 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:10.062144 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:10.063186 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:10.071377 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:10.075654 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:10.083456 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:10.086001 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:10.095075 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:10.118290 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:10.120622 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:10.137230 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:10.173853 29729 consensus_queue.cc:237] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:10.175308 29470 raft_consensus.cc:1261] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:10.175403 31860 raft_consensus.cc:1261] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:10.175606 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef09c96ab4314611a0d51383ad31249b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:10.175727 12800 consensus_queue.cc:1035] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:10.175906 13305 consensus_queue.cc:1035] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:10.178018 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:10.178201 13305 raft_consensus.cc:2932] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:10.178632 29470 raft_consensus.cc:2932] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:10.181494 31860 raft_consensus.cc:2932] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:10.181835 29328 catalog_manager.cc:5484] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:10.204857 29863 consensus_queue.cc:237] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:10.206180 29470 raft_consensus.cc:1261] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:10.206212 32255 raft_consensus.cc:1261] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:10.206310 29798 consensus_peers.cc:485] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4575ec841b443e3bb0820b604e0f074. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:10.206571 13324 consensus_queue.cc:1035] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:10.206761 13029 consensus_queue.cc:1035] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:10.208559 13029 raft_consensus.cc:2932] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:10.208961 32255 raft_consensus.cc:2932] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:10.210572 29470 raft_consensus.cc:2932] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:10.212338 29325 catalog_manager.cc:5484] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:01:10.214458 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.224694 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.233307 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:10.235047 32255 consensus_queue.cc:237] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:10.236351 31860 raft_consensus.cc:1261] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:10.236567 29729 raft_consensus.cc:1261] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:10.236740 10805 consensus_queue.cc:1035] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:10.236923 32189 consensus_peers.cc:485] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ce0cb93decb4313a1677fd5613d1749. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:10.237116 10805 consensus_queue.cc:1035] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:10.239081 13343 raft_consensus.cc:2932] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:10.239557 29732 raft_consensus.cc:2932] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:10.241102 31860 raft_consensus.cc:2932] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:10.243281 29328 catalog_manager.cc:5484] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:10.264418 31860 consensus_queue.cc:237] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
W20240502 14:01:10.265957 31796 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbdb9b4a6c07488f9bc2d15cba59214e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:10.266021 31992 raft_consensus.cc:1261] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:10.266441 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:10.266479 13362 consensus_queue.cc:1035] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:10.297024 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:10.309341 13343 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:10.309448 13343 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:10.309947 13343 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:10.310286 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:10.310444 29470 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:10.312495 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:10.313185 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:10.313254 32188 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:10.313380 13343 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:10.364187 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.366989 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.386432 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.388507 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.415499 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:10.442960 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.446898 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:10.446949 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.466920 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.515370 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.517575 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.519042 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.530871 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.564599 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:10.596545 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.616731 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.630030 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.655092 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:10.659444 13365 ts_tablet_manager.cc:902] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:10.659969 13365 tablet_copy_client.cc:308] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:01:10.660399 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet 4ce0cb93decb4313a1677fd5613d1749 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:59435)
I20240502 14:01:10.660485 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet 4ce0cb93decb4313a1677fd5613d1749 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:59435: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-4ce0cb93decb4313a1677fd5613d1749
I20240502 14:01:10.661449 32275 tablet_copy_source_session.cc:205] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.662267 13365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce0cb93decb4313a1677fd5613d1749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.664124 13365 tablet_copy_client.cc:783] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.664266 13365 tablet_copy_client.cc:647] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.665273 13365 tablet_copy_client.cc:523] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:10.666241 13365 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:01:10.669433 13365 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:10.669685 13365 tablet_bootstrap.cc:492] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:01:10.669862 13365 ts_tablet_manager.cc:1367] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:01:10.670279 13365 raft_consensus.cc:348] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:10.670378 13365 raft_consensus.cc:729] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:01:10.670514 13365 consensus_queue.cc:260] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:10.670773 13365 ts_tablet_manager.cc:1398] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:10.671201 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-4ce0cb93decb4313a1677fd5613d1749 received from {username='slave'} at 127.22.55.132:59435
I20240502 14:01:10.671269 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-4ce0cb93decb4313a1677fd5613d1749 on tablet 4ce0cb93decb4313a1677fd5613d1749 with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
W20240502 14:01:10.709547 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:10.716552 13371 ts_tablet_manager.cc:902] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:10.716984 13371 tablet_copy_client.cc:308] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:01:10.717463 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet ef09c96ab4314611a0d51383ad31249b from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:50019)
I20240502 14:01:10.717588 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet ef09c96ab4314611a0d51383ad31249b from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:50019: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-ef09c96ab4314611a0d51383ad31249b
I20240502 14:01:10.718528 29753 tablet_copy_source_session.cc:205] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.719424 13371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef09c96ab4314611a0d51383ad31249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.721936 13371 tablet_copy_client.cc:783] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.722082 13371 tablet_copy_client.cc:647] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.723202 13371 tablet_copy_client.cc:523] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:10.724383 13371 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:01:10.727483 13371 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:10.727716 13371 tablet_bootstrap.cc:492] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:01:10.727854 13371 ts_tablet_manager.cc:1367] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:01:10.728293 13371 raft_consensus.cc:348] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:10.728412 13371 raft_consensus.cc:729] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:01:10.728551 13371 consensus_queue.cc:260] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:10.728832 13371 ts_tablet_manager.cc:1398] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:10.729846 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-ef09c96ab4314611a0d51383ad31249b received from {username='slave'} at 127.22.55.134:50019
I20240502 14:01:10.729925 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-ef09c96ab4314611a0d51383ad31249b on tablet ef09c96ab4314611a0d51383ad31249b with peer 269b366a09ff435ea8ec8dc01d0f9f6e
W20240502 14:01:10.737743 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.761144 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:10.802102 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:10.806301 13377 ts_tablet_manager.cc:902] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:10.806803 13377 tablet_copy_client.cc:308] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.132:44915
I20240502 14:01:10.807209 29884 tablet_copy_service.cc:140] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Received BeginTabletCopySession request for tablet c4575ec841b443e3bb0820b604e0f074 from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:50041)
I20240502 14:01:10.807291 29884 tablet_copy_service.cc:161] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Beginning new tablet copy session on tablet c4575ec841b443e3bb0820b604e0f074 from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:50041: session id = efdbf7bdfe0b41bdadb1add053b09a7e-c4575ec841b443e3bb0820b604e0f074
I20240502 14:01:10.808007 29884 tablet_copy_source_session.cc:205] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.808831 13377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4575ec841b443e3bb0820b604e0f074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.810950 13377 tablet_copy_client.cc:783] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.811105 13377 tablet_copy_client.cc:647] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.812124 13377 tablet_copy_client.cc:523] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:10.813206 13377 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:01:10.816351 13377 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:10.816629 13377 tablet_bootstrap.cc:492] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:01:10.816776 13377 ts_tablet_manager.cc:1367] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:01:10.817188 13377 raft_consensus.cc:348] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:10.817304 13377 raft_consensus.cc:729] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:01:10.817445 13377 consensus_queue.cc:260] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:10.817749 13377 ts_tablet_manager.cc:1398] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:10.818156 29884 tablet_copy_service.cc:342] P 4aebe6d7a3f34bf7b2bbbfbef1077463: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-c4575ec841b443e3bb0820b604e0f074 received from {username='slave'} at 127.22.55.131:50041
I20240502 14:01:10.818217 29884 tablet_copy_service.cc:433] P 4aebe6d7a3f34bf7b2bbbfbef1077463: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-c4575ec841b443e3bb0820b604e0f074 on tablet c4575ec841b443e3bb0820b604e0f074 with peer efdbf7bdfe0b41bdadb1add053b09a7e
I20240502 14:01:10.855510 13383 ts_tablet_manager.cc:902] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:10.855990 13383 tablet_copy_client.cc:308] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:01:10.856458 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet fbdb9b4a6c07488f9bc2d15cba59214e from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:44365)
I20240502 14:01:10.856556 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet fbdb9b4a6c07488f9bc2d15cba59214e from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:44365: session id = 1d00d3a47c52478894cf8c66959ef7bd-fbdb9b4a6c07488f9bc2d15cba59214e
I20240502 14:01:10.857594 31865 tablet_copy_source_session.cc:205] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:10.858464 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdb9b4a6c07488f9bc2d15cba59214e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:10.860977 13383 tablet_copy_client.cc:783] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:10.861119 13383 tablet_copy_client.cc:647] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:10.862161 13383 tablet_copy_client.cc:523] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:10.863215 13383 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:01:10.865834 13383 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:10.866115 13383 tablet_bootstrap.cc:492] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:01:10.866256 13383 ts_tablet_manager.cc:1367] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:01:10.866719 13383 raft_consensus.cc:348] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:10.867029 13383 raft_consensus.cc:729] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:01:10.867158 13383 consensus_queue.cc:260] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:10.867446 13383 ts_tablet_manager.cc:1398] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:10.867905 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-fbdb9b4a6c07488f9bc2d15cba59214e received from {username='slave'} at 127.22.55.129:44365
I20240502 14:01:10.867977 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-fbdb9b4a6c07488f9bc2d15cba59214e on tablet fbdb9b4a6c07488f9bc2d15cba59214e with peer 1d00d3a47c52478894cf8c66959ef7bd
W20240502 14:01:10.916672 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:11.037272 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:11.177728 31990 raft_consensus.cc:1203] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:01:11.254417 29863 raft_consensus.cc:1203] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:01:11.270133 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:11.311233 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:11.319876 29729 raft_consensus.cc:1203] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:01:11.384621 29469 raft_consensus.cc:1203] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:01:11.385294 12460 raft_consensus.cc:2932] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:11.386173 29469 raft_consensus.cc:2932] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:11.386976 12460 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:47426 (request call id 0) took 1122 ms. Trace:
W20240502 14:01:11.387428 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:11.387058 12460 rpcz_store.cc:276] 0502 14:01:10.264067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:01:10.264121 (+ 54us) service_pool.cc:225] Handling call
0502 14:01:10.265062 (+ 941us) log.cc:842] Serialized 498 byte log entry
0502 14:01:11.386961 (+1121899us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":50,"mutex_wait_us":17,"raft.queue_time_us":243,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":328,"spinlock_wait_cycles":6656,"thread_start_us":117,"threads_started":2,"wal-append.queue_time_us":128,"wal-append.run_cpu_time_us":371,"wal-append.run_wall_time_us":1000501}
I20240502 14:01:11.391557 31990 raft_consensus.cc:2932] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:11.393316 29323 catalog_manager.cc:5484] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 2, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:01:11.448292 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:11.529682 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:11.532243 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:11.538617 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:11.542508 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:11.542623 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:11.563823 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:11.580417 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:11.694471 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:11.695724 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:11.721653 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:11.737565 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:11.770502 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:11.776010 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:11.781814 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:11.782467 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:11.782716 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:11.803264 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:11.808401 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:11.808710 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:11.864643 32255 consensus_queue.cc:237] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:11.865823 29864 raft_consensus.cc:1261] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:11.865952 29729 raft_consensus.cc:1261] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:11.866283 13426 consensus_queue.cc:1035] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:11.866472 13343 consensus_queue.cc:1035] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:11.868165 13426 raft_consensus.cc:2932] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:11.868758 29864 raft_consensus.cc:2932] T 4ce0cb93decb4313a1677fd5613d1749 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:11.868737 29729 raft_consensus.cc:2932] T 4ce0cb93decb4313a1677fd5613d1749 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:01:11.870671 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:11.871954 29329 catalog_manager.cc:5484] T 4ce0cb93decb4313a1677fd5613d1749 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index 2 to 3, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:11.874931 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce0cb93decb4313a1677fd5613d1749 with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43670
I20240502 14:01:11.875334 13536 tablet_replica.cc:330] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:01:11.875520 13536 raft_consensus.cc:2229] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:11.875720 13536 raft_consensus.cc:2258] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:11.876207 13536 ts_tablet_manager.cc:1875] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:11.878610 13536 ts_tablet_manager.cc:1888] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:11.878717 13536 log.cc:1196] T 4ce0cb93decb4313a1677fd5613d1749 P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/4ce0cb93decb4313a1677fd5613d1749
I20240502 14:01:11.879683 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet 4ce0cb93decb4313a1677fd5613d1749 (table rebalancer_extra_table_14 [id=2f82b6ee513f438db639f6570a641907]) successfully deleted
W20240502 14:01:11.906425 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:11.916404 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:11.961738 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:12.024955 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:12.025077 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:12.025458 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.031634 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:12.034997 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:12.037297 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:12.039567 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:12.039650 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:12.057106 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:12.059046 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.059376 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.067230 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:12.105561 13343 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:12.105695 13343 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:12.106192 13343 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:12.106469 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:12.106596 29864 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:12.106871 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:12.107141 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:12.107203 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:12.107455 13343 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:12.125869 29729 consensus_queue.cc:237] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:12.127115 31860 raft_consensus.cc:1261] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:12.127338 31992 raft_consensus.cc:1261] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:12.127692 13390 consensus_queue.cc:1035] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:12.127988 13305 consensus_queue.cc:1035] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:01:12.129673 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:12.129927 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:12.130540 13305 raft_consensus.cc:2932] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:12.130859 31992 raft_consensus.cc:2932] T ef09c96ab4314611a0d51383ad31249b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:12.135377 29324 catalog_manager.cc:5484] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
W20240502 14:01:12.135951 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:12.138130 31860 raft_consensus.cc:2932] T ef09c96ab4314611a0d51383ad31249b P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:12.139719 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet ef09c96ab4314611a0d51383ad31249b with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57986
I20240502 14:01:12.140087 13607 tablet_replica.cc:330] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:01:12.140234 13607 raft_consensus.cc:2229] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:12.140349 13607 raft_consensus.cc:2258] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:12.140790 13607 ts_tablet_manager.cc:1875] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:12.144733 13607 ts_tablet_manager.cc:1888] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:12.144809 13607 log.cc:1196] T ef09c96ab4314611a0d51383ad31249b P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef09c96ab4314611a0d51383ad31249b
I20240502 14:01:12.145376 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet ef09c96ab4314611a0d51383ad31249b (table rebalancer_extra_table_52 [id=a50b51dee63b4f0384e3b7ec0300c4d0]) successfully deleted
W20240502 14:01:12.154914 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.162221 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.164824 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.219026 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.274122 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:12.274968 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:12.278836 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:12.281908 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:12.286831 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:12.288306 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:12.288857 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:12.301784 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.314687 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:12.341903 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:12.397641 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.410604 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.461794 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.485626 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:12.489883 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:12.492290 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:12.494930 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:12.504323 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:12.505820 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:12.513442 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:12.516094 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:12.518983 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:12.521948 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.535687 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.548421 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.556422 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.585655 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.602505 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.603469 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:12.625933 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:12.642060 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.678633 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:12.702903 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:12.703190 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:12.707929 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:12.708956 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:12.710608 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:12.749127 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:12.759510 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:12.761801 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:12.767499 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:12.775715 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:12.810575 31990 consensus_queue.cc:237] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:12.811775 31861 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:12.811808 29469 raft_consensus.cc:1261] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:12.812196 13671 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:12.812392 13389 consensus_queue.cc:1035] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:12.813972 13780 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:12.814319 29469 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:12.814728 31861 raft_consensus.cc:2932] T 1fc14b4bd42841df91972e0c5d02faa4 P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:12.819864 29321 catalog_manager.cc:5484] T 1fc14b4bd42841df91972e0c5d02faa4 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 6 to 8, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:12.823535 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc14b4bd42841df91972e0c5d02faa4 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54432
I20240502 14:01:12.824564 13791 tablet_replica.cc:330] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:01:12.825126 13791 raft_consensus.cc:2229] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:12.825325 13791 raft_consensus.cc:2258] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:12.825930 13791 ts_tablet_manager.cc:1875] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:12.830037 13791 ts_tablet_manager.cc:1888] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:01:12.830358 13791 log.cc:1196] T 1fc14b4bd42841df91972e0c5d02faa4 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/1fc14b4bd42841df91972e0c5d02faa4
I20240502 14:01:12.831156 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet 1fc14b4bd42841df91972e0c5d02faa4 (table rebalance_test_table_3 [id=89f38791b32c42d0bf1cff20ca03b343]) successfully deleted
W20240502 14:01:12.893963 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.895444 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.918246 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:12.953101 13426 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:12.953222 13426 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:12.953732 13426 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:12.957039 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:12.957197 29469 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:12.957870 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
I20240502 14:01:12.958889 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:12.959772 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:12.965220 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:12.966282 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:12.966459 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:12.966536 32188 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:12.966871 13343 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:12.968878 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:12.971817 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:12.990294 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.996357 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.996460 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:12.996574 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:13.002590 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:13.006007 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.013996 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.018474 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.019016 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:13.033354 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.034631 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:13.060353 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.067476 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:13.116022 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.119558 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240502 14:01:13.141593 31794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:01:13.147480 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:01:13.185714 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:01:13.192688 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:13.194227 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:13.204326 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.205638 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 95 similar messages]
I20240502 14:01:13.207093 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:13.208528 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:13.213687 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:13.214500 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 14:01:13.215193 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:13.215229 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:13.217761 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:13.334519 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.341634 29535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:01:13.401453 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:13.405329 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:13.408229 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:13.414005 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:13.419294 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:13.420971 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:13.429579 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:13.430186 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:13.492095 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:13.613390 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:13.618407 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:13.621960 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:13.623579 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:13.637710 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:13.639798 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:13.642181 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:13.655865 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:13.838199 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:13.839607 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:13.841923 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:13.842105 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:13.847000 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:13.847887 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:13.852596 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:13.932099 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:13.996438 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.056025 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:14.065184 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:14.069157 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:14.083029 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:14.083323 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:14.088261 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:14.099045 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:14.125119 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:14.135422 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.186015 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:14.186478 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.216734 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:14.294756 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:14.300490 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:14.307169 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:14.318096 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:14.321992 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:14.330873 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:14.337109 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:14.343688 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:14.358192 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.375658 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:14.404660 29863 consensus_queue.cc:237] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:14.405812 29732 raft_consensus.cc:1261] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:14.405915 29469 raft_consensus.cc:1261] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:14.406265 13447 consensus_queue.cc:1035] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:14.406479 14149 consensus_queue.cc:1035] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:14.408259 14149 raft_consensus.cc:2932] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:14.408593 29469 raft_consensus.cc:2932] T c4575ec841b443e3bb0820b604e0f074 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:14.410602 29732 raft_consensus.cc:2932] T c4575ec841b443e3bb0820b604e0f074 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:14.412441 29323 catalog_manager.cc:5484] T c4575ec841b443e3bb0820b604e0f074 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:14.417254 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet c4575ec841b443e3bb0820b604e0f074 with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:01:14.417618 14188 tablet_replica.cc:330] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:01:14.417759 14188 raft_consensus.cc:2229] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:14.418102 14188 raft_consensus.cc:2258] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:14.418551 14188 ts_tablet_manager.cc:1875] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:14.421289 14188 ts_tablet_manager.cc:1888] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:14.421367 14188 log.cc:1196] T c4575ec841b443e3bb0820b604e0f074 P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/c4575ec841b443e3bb0820b604e0f074
I20240502 14:01:14.422144 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet c4575ec841b443e3bb0820b604e0f074 (table rebalancer_extra_table_24 [id=7df2b4c1d7984193b964d003737f7945]) successfully deleted
W20240502 14:01:14.467271 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:14.507462 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.512763 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:14.520143 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.526072 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:14.528853 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.550455 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:14.552920 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:14.559336 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:14.559906 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:14.562302 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:14.563709 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:14.565738 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:14.568280 14149 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:14.568390 14149 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:14.568873 14149 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
W20240502 14:01:14.570443 29799 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
I20240502 14:01:14.571519 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:14.571744 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:14.577824 29797 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:14.577919 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:14.578078 14149 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:14.578035 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:14.578186 32255 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
W20240502 14:01:14.604611 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.612329 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.635578 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.635699 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:14.675954 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.751801 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.758041 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:14.761276 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:14.761958 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:14.762364 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:14.765517 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:14.778189 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:14.781502 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:14.785567 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:14.789075 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:14.789458 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.790417 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:14.810751 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:14.852728 29732 consensus_queue.cc:237] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:14.854197 29470 raft_consensus.cc:1261] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:14.854519 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25d043c246e94f3090295583133ca0cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.854611 32253 raft_consensus.cc:1261] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.854604 13997 consensus_queue.cc:1035] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:14.855264 13997 consensus_queue.cc:1035] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:14.856988 13715 raft_consensus.cc:2932] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
W20240502 14:01:14.857328 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:14.857543 32253 raft_consensus.cc:2932] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:14.861256 29325 catalog_manager.cc:5484] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:14.861327 29470 raft_consensus.cc:2932] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
W20240502 14:01:14.871845 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:14.878016 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:14.882460 29600 consensus_queue.cc:237] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:14.884204 29470 raft_consensus.cc:1261] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:14.884442 29536 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f28e9f5068aa48b5a87841bdf720d9e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:14.884572 29732 raft_consensus.cc:1261] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.884624 14325 consensus_queue.cc:1035] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:14.886914 14325 consensus_queue.cc:1035] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:14.888612 13722 raft_consensus.cc:2932] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:14.890651 29470 raft_consensus.cc:2932] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:14.890657 29732 raft_consensus.cc:2932] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:14.892421 29329 catalog_manager.cc:5484] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:01:14.929992 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:14.992770 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:14.994275 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.000306 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:15.026892 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:15.030032 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:15.038022 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:15.041083 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:15.044782 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.047228 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:15.067339 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.070219 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:15.071660 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:15.079355 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:15.122859 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:15.133349 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:15.175738 13343 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:15.175861 13343 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:15.176337 13343 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:15.176883 29470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:15.177021 29470 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:15.179960 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:15.180060 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:15.180104 32188 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:15.180245 13343 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:15.183053 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:15.192066 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:15.198374 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.214183 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.255082 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:15.258384 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:15.265046 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:15.268718 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:15.275293 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:15.275447 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.283334 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:15.292688 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:15.324465 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.347250 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.359054 14446 ts_tablet_manager.cc:902] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:15.359993 14446 tablet_copy_client.cc:308] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.131:38105
I20240502 14:01:15.360481 29753 tablet_copy_service.cc:140] P efdbf7bdfe0b41bdadb1add053b09a7e: Received BeginTabletCopySession request for tablet 25d043c246e94f3090295583133ca0cd from peer d6e2e2c10def4331bec17bd110c9b423 ({username='slave'} at 127.22.55.130:60787)
I20240502 14:01:15.360586 29753 tablet_copy_service.cc:161] P efdbf7bdfe0b41bdadb1add053b09a7e: Beginning new tablet copy session on tablet 25d043c246e94f3090295583133ca0cd from peer d6e2e2c10def4331bec17bd110c9b423 at {username='slave'} at 127.22.55.130:60787: session id = d6e2e2c10def4331bec17bd110c9b423-25d043c246e94f3090295583133ca0cd
I20240502 14:01:15.361528 29753 tablet_copy_source_session.cc:205] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:15.362452 14446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d043c246e94f3090295583133ca0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.364454 14446 tablet_copy_client.cc:783] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:15.364609 14446 tablet_copy_client.cc:647] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:15.365620 14446 tablet_copy_client.cc:523] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:15.366712 14446 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: Bootstrap starting.
I20240502 14:01:15.371062 14446 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:15.371299 14446 tablet_bootstrap.cc:492] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: Bootstrap complete.
I20240502 14:01:15.371462 14446 ts_tablet_manager.cc:1367] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:01:15.371821 14446 raft_consensus.cc:348] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:15.371929 14446 raft_consensus.cc:729] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e2e2c10def4331bec17bd110c9b423, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.372083 14446 consensus_queue.cc:260] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:15.373082 14446 ts_tablet_manager.cc:1398] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:15.373531 29753 tablet_copy_service.cc:342] P efdbf7bdfe0b41bdadb1add053b09a7e: Request end of tablet copy session d6e2e2c10def4331bec17bd110c9b423-25d043c246e94f3090295583133ca0cd received from {username='slave'} at 127.22.55.130:60787
I20240502 14:01:15.373615 29753 tablet_copy_service.cc:433] P efdbf7bdfe0b41bdadb1add053b09a7e: ending tablet copy session d6e2e2c10def4331bec17bd110c9b423-25d043c246e94f3090295583133ca0cd on tablet 25d043c246e94f3090295583133ca0cd with peer d6e2e2c10def4331bec17bd110c9b423
W20240502 14:01:15.405308 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:15.406455 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.442828 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.460371 14482 ts_tablet_manager.cc:902] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:15.460916 14482 tablet_copy_client.cc:308] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.130:40041
I20240502 14:01:15.461779 29622 tablet_copy_service.cc:140] P d6e2e2c10def4331bec17bd110c9b423: Received BeginTabletCopySession request for tablet f28e9f5068aa48b5a87841bdf720d9e0 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:53217)
I20240502 14:01:15.461871 29622 tablet_copy_service.cc:161] P d6e2e2c10def4331bec17bd110c9b423: Beginning new tablet copy session on tablet f28e9f5068aa48b5a87841bdf720d9e0 from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:53217: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-f28e9f5068aa48b5a87841bdf720d9e0
I20240502 14:01:15.462728 29622 tablet_copy_source_session.cc:205] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:15.463707 14482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28e9f5068aa48b5a87841bdf720d9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.466426 14482 tablet_copy_client.cc:783] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:15.466580 14482 tablet_copy_client.cc:647] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:15.469012 14482 tablet_copy_client.cc:523] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:15.472429 14482 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:01:15.481158 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:15.489171 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:15.499871 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:15.503314 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:15.508700 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:15.517491 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:15.527271 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:15.551033 14482 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:15.551504 14482 tablet_bootstrap.cc:492] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:01:15.551718 14482 ts_tablet_manager.cc:1367] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20240502 14:01:15.552232 14482 raft_consensus.cc:348] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:15.552367 14482 raft_consensus.cc:729] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.552502 14482 consensus_queue.cc:260] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:15.552857 14482 ts_tablet_manager.cc:1398] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.554373 29622 tablet_copy_service.cc:342] P d6e2e2c10def4331bec17bd110c9b423: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-f28e9f5068aa48b5a87841bdf720d9e0 received from {username='slave'} at 127.22.55.132:53217
I20240502 14:01:15.554463 29622 tablet_copy_service.cc:433] P d6e2e2c10def4331bec17bd110c9b423: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-f28e9f5068aa48b5a87841bdf720d9e0 on tablet f28e9f5068aa48b5a87841bdf720d9e0 with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
W20240502 14:01:15.555881 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.557993 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.558578 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.576988 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:15.591252 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.646349 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.693213 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.701391 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:15.716301 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:15.725189 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:15.727929 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:15.732687 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:15.739984 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:15.743057 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.744464 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.749029 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:15.775444 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:15.852203 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:15.858570 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.861757 29863 raft_consensus.cc:1203] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:01:15.869242 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:15.870636 29600 raft_consensus.cc:1203] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:01:15.931629 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:15.939046 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:15.939924 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:15.943284 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:15.952445 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:15.952816 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:15.981251 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:16.024032 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:16.060765 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:16.158614 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:16.160101 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:16.176975 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:16.179610 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:16.189025 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:16.201373 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:16.205756 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:16.296463 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.309589 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:16.384013 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:16.386634 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:16.389381 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:16.390810 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:16.393404 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:16.395364 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:16.403825 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:16.513311 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.547509 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:16.590200 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:16.594056 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:16.598793 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:16.598793 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:16.602492 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:16.608764 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:16.613693 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:16.640439 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:16.648368 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.655433 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.668371 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.686909 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.754696 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.792042 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:16.807135 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:16.809935 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:16.810017 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:16.814392 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:16.814589 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:16.815462 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:16.825642 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:16.839746 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:16.865154 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.915306 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.921125 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:16.955628 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.957193 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:16.978996 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.004426 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:17.015450 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:17.024104 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:17.026434 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:17.026611 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:17.029429 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:17.038197 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:17.038755 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:17.081688 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:17.096302 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:17.218730 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:17.237593 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:17.238642 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:17.240623 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:17.241101 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:17.245801 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:17.246949 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:17.247539 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:17.251865 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:17.281816 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:17.285351 14855 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:17.285487 14855 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:17.286092 14855 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:17.286371 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:17.286505 32255 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:17.287672 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:17.287783 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:17.287858 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:17.288064 14855 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:17.300570 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:17.321102 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.330467 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.345479 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:17.348767 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:17.385296 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.387996 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:17.402192 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.403178 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.437263 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:17.460294 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:17.464280 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:17.464278 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:17.473572 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:17.475270 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:17.477933 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:17.481006 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:17.483223 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:17.483783 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:17.500500 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.556176 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:17.567898 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.576643 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.620441 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.639034 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.643891 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:17.662346 14913 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:17.662487 14913 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:17.663039 14913 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:17.663882 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:17.664022 32255 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:17.665190 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:17.666164 29799 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:17.669555 29797 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:17.669628 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:17.670801 14913 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:17.672065 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:17.674615 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:17.675547 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:17.680775 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:17.684531 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:17.686865 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:17.733116 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:17.733197 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:17.764503 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:17.774011 31859 consensus_queue.cc:237] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:17.775585 32253 raft_consensus.cc:1261] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.775585 31992 raft_consensus.cc:1261] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer f0e01c4f581843e9ad4ced97f0072dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:17.775631 31795 consensus_peers.cc:485] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Couldn't send request to peer 4aebe6d7a3f34bf7b2bbbfbef1077463. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea104c9432a7416f87760e7f6458d6cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:17.776002 14871 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:17.776192 14608 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:17.778275 14871 raft_consensus.cc:2932] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:17.778710 31992 raft_consensus.cc:2932] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:17.780397 32253 raft_consensus.cc:2932] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:17.782076 29326 catalog_manager.cc:5484] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 reported cstate change: config changed from index -1 to 2, VOTER 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132) added. New cstate: current_term: 1 leader_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:01:17.795519 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:17.804764 31992 consensus_queue.cc:237] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:17.806202 29863 raft_consensus.cc:1261] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240502 14:01:17.806268 31927 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Couldn't send request to peer 1d00d3a47c52478894cf8c66959ef7bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeab77cba6cf431e89141d4154c41a9b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:17.806588 15075 consensus_queue.cc:1035] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:01:17.806764 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:17.821725 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.846753 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:17.871173 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:17.887709 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:17.961699 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.030308 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.089658 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.120951 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240502 14:01:18.140934 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.148347 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:01:18.149606 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:18.165565 31794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:01:18.166119 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.212203 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.234752 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:01:18.244078 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:01:18.275575 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:18.282352 15077 ts_tablet_manager.cc:902] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: Initiating tablet copy from peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:18.282819 15077 tablet_copy_client.cc:308] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.133:39647
I20240502 14:01:18.283326 31865 tablet_copy_service.cc:140] P f0e01c4f581843e9ad4ced97f0072dc4: Received BeginTabletCopySession request for tablet ea104c9432a7416f87760e7f6458d6cb from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 ({username='slave'} at 127.22.55.132:41401)
I20240502 14:01:18.283445 31865 tablet_copy_service.cc:161] P f0e01c4f581843e9ad4ced97f0072dc4: Beginning new tablet copy session on tablet ea104c9432a7416f87760e7f6458d6cb from peer 4aebe6d7a3f34bf7b2bbbfbef1077463 at {username='slave'} at 127.22.55.132:41401: session id = 4aebe6d7a3f34bf7b2bbbfbef1077463-ea104c9432a7416f87760e7f6458d6cb
I20240502 14:01:18.284379 31865 tablet_copy_source_session.cc:205] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:18.285173 15077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea104c9432a7416f87760e7f6458d6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.287257 15077 tablet_copy_client.cc:783] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:18.287431 15077 tablet_copy_client.cc:647] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:18.288718 15077 tablet_copy_client.cc:523] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:18.289912 15077 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap starting.
I20240502 14:01:18.292763 15077 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:18.293047 15077 tablet_bootstrap.cc:492] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: Bootstrap complete.
I20240502 14:01:18.293366 15077 ts_tablet_manager.cc:1367] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.003s
I20240502 14:01:18.294054 15077 raft_consensus.cc:348] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:18.294180 15077 raft_consensus.cc:729] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.294310 15077 consensus_queue.cc:260] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:18.294608 15077 ts_tablet_manager.cc:1398] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:18.295053 31865 tablet_copy_service.cc:342] P f0e01c4f581843e9ad4ced97f0072dc4: Request end of tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-ea104c9432a7416f87760e7f6458d6cb received from {username='slave'} at 127.22.55.132:41401
I20240502 14:01:18.295137 31865 tablet_copy_service.cc:433] P f0e01c4f581843e9ad4ced97f0072dc4: ending tablet copy session 4aebe6d7a3f34bf7b2bbbfbef1077463-ea104c9432a7416f87760e7f6458d6cb on tablet ea104c9432a7416f87760e7f6458d6cb with peer 4aebe6d7a3f34bf7b2bbbfbef1077463
W20240502 14:01:18.316368 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:01:18.334981 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.360877 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.361532 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:18.363550 15084 ts_tablet_manager.cc:902] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:18.363970 15084 tablet_copy_client.cc:308] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.134:44137
I20240502 14:01:18.364418 32012 tablet_copy_service.cc:140] P 269b366a09ff435ea8ec8dc01d0f9f6e: Received BeginTabletCopySession request for tablet aeab77cba6cf431e89141d4154c41a9b from peer 1d00d3a47c52478894cf8c66959ef7bd ({username='slave'} at 127.22.55.129:56031)
I20240502 14:01:18.364519 32012 tablet_copy_service.cc:161] P 269b366a09ff435ea8ec8dc01d0f9f6e: Beginning new tablet copy session on tablet aeab77cba6cf431e89141d4154c41a9b from peer 1d00d3a47c52478894cf8c66959ef7bd at {username='slave'} at 127.22.55.129:56031: session id = 1d00d3a47c52478894cf8c66959ef7bd-aeab77cba6cf431e89141d4154c41a9b
I20240502 14:01:18.365481 32012 tablet_copy_source_session.cc:205] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:18.366324 15084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeab77cba6cf431e89141d4154c41a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.368332 15084 tablet_copy_client.cc:783] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:18.368490 15084 tablet_copy_client.cc:647] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:18.369531 15084 tablet_copy_client.cc:523] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:18.370612 15084 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap starting.
I20240502 14:01:18.373663 15084 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:18.373888 15084 tablet_bootstrap.cc:492] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: Bootstrap complete.
I20240502 14:01:18.374027 15084 ts_tablet_manager.cc:1367] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:01:18.374399 15084 raft_consensus.cc:348] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:18.374608 15084 raft_consensus.cc:729] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.374713 15084 consensus_queue.cc:260] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:18.374995 15084 ts_tablet_manager.cc:1398] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:18.375481 32012 tablet_copy_service.cc:342] P 269b366a09ff435ea8ec8dc01d0f9f6e: Request end of tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-aeab77cba6cf431e89141d4154c41a9b received from {username='slave'} at 127.22.55.129:56031
I20240502 14:01:18.375557 32012 tablet_copy_service.cc:433] P 269b366a09ff435ea8ec8dc01d0f9f6e: ending tablet copy session 1d00d3a47c52478894cf8c66959ef7bd-aeab77cba6cf431e89141d4154c41a9b on tablet aeab77cba6cf431e89141d4154c41a9b with peer 1d00d3a47c52478894cf8c66959ef7bd
W20240502 14:01:18.387333 29535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:01:18.392504 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.414014 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.490952 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:18.502012 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.597654 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:18.879602 29864 raft_consensus.cc:1203] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:01:18.902529 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:18.908668 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:18.951465 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:18.961874 29469 raft_consensus.cc:1203] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 14:01:18.962641 13780 raft_consensus.cc:2932] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:18.963445 29469 raft_consensus.cc:2932] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:18.966951 13780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:58054 (request call id 0) took 1162 ms. Trace:
I20240502 14:01:18.969014 29327 catalog_manager.cc:5484] T aeab77cba6cf431e89141d4154c41a9b P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index -1 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:18.967046 13780 rpcz_store.cc:276] 0502 14:01:17.804218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:01:17.804437 (+ 219us) service_pool.cc:225] Handling call
0502 14:01:17.805375 (+ 938us) log.cc:842] Serialized 498 byte log entry
0502 14:01:18.966936 (+1161561us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":69,"mutex_wait_us":2,"raft.queue_time_us":244,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":318,"spinlock_wait_cycles":12928,"thread_start_us":106,"threads_started":2,"wal-append.queue_time_us":123,"wal-append.run_cpu_time_us":287,"wal-append.run_wall_time_us":1000351}
I20240502 14:01:18.969952 29864 raft_consensus.cc:2932] T aeab77cba6cf431e89141d4154c41a9b P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:01:18.988013 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:18.994485 32253 consensus_queue.cc:237] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
W20240502 14:01:18.996201 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137): Couldn't send request to peer 269b366a09ff435ea8ec8dc01d0f9f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93e9d6f5e9b84267afa5e019f6bb8c08. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:18.996276 29732 raft_consensus.cc:1261] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240502 14:01:18.996796 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:18.996846 15083 consensus_queue.cc:1035] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:01:19.010275 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.108836 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.110560 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.151542 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:19.161085 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.218472 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.326449 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.331666 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.340730 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.363332 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:19.377573 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.481206 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:19.541662 15122 ts_tablet_manager.cc:902] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:19.542176 15122 tablet_copy_client.cc:308] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:01:19.542565 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet 93e9d6f5e9b84267afa5e019f6bb8c08 from peer 269b366a09ff435ea8ec8dc01d0f9f6e ({username='slave'} at 127.22.55.134:60657)
I20240502 14:01:19.542650 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet 93e9d6f5e9b84267afa5e019f6bb8c08 from peer 269b366a09ff435ea8ec8dc01d0f9f6e at {username='slave'} at 127.22.55.134:60657: session id = 269b366a09ff435ea8ec8dc01d0f9f6e-93e9d6f5e9b84267afa5e019f6bb8c08
I20240502 14:01:19.543591 32275 tablet_copy_source_session.cc:205] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:19.544378 15122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e9d6f5e9b84267afa5e019f6bb8c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:19.546829 15122 tablet_copy_client.cc:783] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:19.547014 15122 tablet_copy_client.cc:647] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:19.548054 15122 tablet_copy_client.cc:523] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:19.549078 15122 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap starting.
I20240502 14:01:19.552495 15122 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:19.552865 15122 tablet_bootstrap.cc:492] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: Bootstrap complete.
I20240502 14:01:19.553058 15122 ts_tablet_manager.cc:1367] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.005s
I20240502 14:01:19.553436 15122 raft_consensus.cc:348] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:19.553650 15122 raft_consensus.cc:729] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Initialized, Role: FOLLOWER
I20240502 14:01:19.553745 15122 consensus_queue.cc:260] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:19.554025 15122 ts_tablet_manager.cc:1398] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:19.554559 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-93e9d6f5e9b84267afa5e019f6bb8c08 received from {username='slave'} at 127.22.55.134:60657
I20240502 14:01:19.554625 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session 269b366a09ff435ea8ec8dc01d0f9f6e-93e9d6f5e9b84267afa5e019f6bb8c08 on tablet 93e9d6f5e9b84267afa5e019f6bb8c08 with peer 269b366a09ff435ea8ec8dc01d0f9f6e
I20240502 14:01:19.603688 14983 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:19.603816 14983 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:19.604445 14983 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
W20240502 14:01:19.605091 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:19.605645 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:19.605636 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:19.605772 32253 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:19.606230 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:19.606303 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:19.606483 14983 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:19.679775 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.694703 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.715114 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:19.725224 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.725842 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.736667 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.761372 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.799674 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.828032 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:19.833230 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.865953 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:19.902706 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:19.903762 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.913118 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.914901 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.968101 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.974068 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:19.990736 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:20.018934 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:20.061789 31992 raft_consensus.cc:1203] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 14:01:20.062393 15120 raft_consensus.cc:2932] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:20.062796 31992 raft_consensus.cc:2932] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:20.064095 15120 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54938 (request call id 0) took 1070 ms. Trace:
I20240502 14:01:20.064186 15120 rpcz_store.cc:276] 0502 14:01:18.994089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:01:18.994138 (+ 49us) service_pool.cc:225] Handling call
0502 14:01:18.995242 (+ 1104us) log.cc:842] Serialized 498 byte log entry
0502 14:01:20.064077 (+1068835us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":56,"raft.queue_time_us":253,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":338,"spinlock_wait_cycles":148096,"thread_start_us":111,"threads_started":2,"wal-append.queue_time_us":198,"wal-append.run_cpu_time_us":455,"wal-append.run_wall_time_us":1001721}
I20240502 14:01:20.067266 29328 catalog_manager.cc:5484] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:20.068186 29732 raft_consensus.cc:2932] T 93e9d6f5e9b84267afa5e019f6bb8c08 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
W20240502 14:01:20.076385 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.083277 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:20.090988 32253 consensus_queue.cc:237] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:20.092407 29864 raft_consensus.cc:1261] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:01:20.092535 32189 consensus_peers.cc:485] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 -> Peer efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): Couldn't send request to peer efdbf7bdfe0b41bdadb1add053b09a7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddb52469e62d4e6ab5bcc90cea577c9c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:20.092561 29470 raft_consensus.cc:1261] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:20.092906 15120 consensus_queue.cc:1035] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:20.093088 15083 consensus_queue.cc:1035] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:01:20.094995 15120 raft_consensus.cc:2932] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:20.096975 29470 raft_consensus.cc:2932] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:20.098310 29326 catalog_manager.cc:5484] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 reported cstate change: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:20.098352 29864 raft_consensus.cc:2932] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:01:20.104161 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:20.115971 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:20.116657 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.124302 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.134935 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.173617 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:20.180033 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.182428 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.220667 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:20.244423 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:20.248708 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:20.249974 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:20.254642 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:20.255666 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:20.261489 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:20.267576 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:20.281805 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.297962 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.322515 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.347056 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:20.467057 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:20.474555 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:20.475116 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:20.478863 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:20.480201 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:20.480520 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:20.495201 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 14:01:20.504729 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:20.552739 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.568519 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:20.588575 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.648013 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:20.681030 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:20.681565 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:20.682837 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:20.686460 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:20.686832 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:20.695919 15326 ts_tablet_manager.cc:902] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: Initiating tablet copy from peer 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:20.696483 15326 tablet_copy_client.cc:308] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Beginning tablet copy session from remote peer at address 127.22.55.136:44201
I20240502 14:01:20.704794 32275 tablet_copy_service.cc:140] P 8973f2527acb48f5ad36589652de73e1: Received BeginTabletCopySession request for tablet ddb52469e62d4e6ab5bcc90cea577c9c from peer efdbf7bdfe0b41bdadb1add053b09a7e ({username='slave'} at 127.22.55.131:39103)
I20240502 14:01:20.704893 32275 tablet_copy_service.cc:161] P 8973f2527acb48f5ad36589652de73e1: Beginning new tablet copy session on tablet ddb52469e62d4e6ab5bcc90cea577c9c from peer efdbf7bdfe0b41bdadb1add053b09a7e at {username='slave'} at 127.22.55.131:39103: session id = efdbf7bdfe0b41bdadb1add053b09a7e-ddb52469e62d4e6ab5bcc90cea577c9c
I20240502 14:01:20.705832 32275 tablet_copy_source_session.cc:205] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:20.706885 15326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb52469e62d4e6ab5bcc90cea577c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:20.711416 15326 tablet_copy_client.cc:783] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:20.711575 15326 tablet_copy_client.cc:647] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:20.713207 15326 tablet_copy_client.cc:523] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:20.715889 15326 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap starting.
I20240502 14:01:20.716763 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:20.724623 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:20.725374 15326 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:20.725653 15326 tablet_bootstrap.cc:492] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: Bootstrap complete.
I20240502 14:01:20.725817 15326 ts_tablet_manager.cc:1367] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 14:01:20.726311 15326 raft_consensus.cc:348] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:20.726433 15326 raft_consensus.cc:729] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Initialized, Role: FOLLOWER
I20240502 14:01:20.726568 15326 consensus_queue.cc:260] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:20.729533 15326 ts_tablet_manager.cc:1398] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:20.731227 32275 tablet_copy_service.cc:342] P 8973f2527acb48f5ad36589652de73e1: Request end of tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-ddb52469e62d4e6ab5bcc90cea577c9c received from {username='slave'} at 127.22.55.131:39103
I20240502 14:01:20.731302 32275 tablet_copy_service.cc:433] P 8973f2527acb48f5ad36589652de73e1: ending tablet copy session efdbf7bdfe0b41bdadb1add053b09a7e-ddb52469e62d4e6ab5bcc90cea577c9c on tablet ddb52469e62d4e6ab5bcc90cea577c9c with peer efdbf7bdfe0b41bdadb1add053b09a7e
W20240502 14:01:20.743754 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.777479 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.828959 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.853449 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.862404 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.871019 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:20.895313 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:20.899834 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:20.906831 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:20.911550 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:20.912441 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:20.917738 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:20.920189 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:20.932603 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 14:01:20.968868 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:20.975991 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:21.091874 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:21.129037 29843 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 14:01:21.135241 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:21.138886 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:21.141433 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:21.143504 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:21.151152 31840 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:01:21.165127 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:21.211117 31859 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb"
dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39900
I20240502 14:01:21.211290 31859 raft_consensus.cc:595] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:21.231674 14871 raft_consensus.cc:979] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: : Instructing follower 4aebe6d7a3f34bf7b2bbbfbef1077463 to start an election
I20240502 14:01:21.231803 14871 raft_consensus.cc:1067] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Signalling peer 4aebe6d7a3f34bf7b2bbbfbef1077463 to start an election
I20240502 14:01:21.232317 29864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb"
dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
from {username='slave'} at 127.22.55.133:44851
I20240502 14:01:21.232450 29864 raft_consensus.cc:482] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:21.232498 29864 raft_consensus.cc:3037] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:21.233551 29864 raft_consensus.cc:504] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:21.234072 29864 leader_election.cc:288] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:21.234388 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f0e01c4f581843e9ad4ced97f0072dc4"
I20240502 14:01:21.234411 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:01:21.234513 32253 raft_consensus.cc:3037] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:21.234513 31859 raft_consensus.cc:3032] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:21.234553 31859 raft_consensus.cc:729] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f0e01c4f581843e9ad4ced97f0072dc4, State: Running, Role: LEADER
I20240502 14:01:21.234548 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea104c9432a7416f87760e7f6458d6cb" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:21.234658 31992 raft_consensus.cc:3037] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:21.234733 31859 consensus_queue.cc:260] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:21.234917 31859 raft_consensus.cc:3037] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:21.235710 32253 raft_consensus.cc:2454] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:01:21.235817 31992 raft_consensus.cc:2454] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:01:21.235967 31859 raft_consensus.cc:2454] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:01:21.236213 29797 leader_election.cc:302] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4; no voters:
I20240502 14:01:21.236622 15314 raft_consensus.cc:2781] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:21.236788 15314 raft_consensus.cc:686] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:21.236936 15314 consensus_queue.cc:237] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:21.240002 29324 catalog_manager.cc:5484] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:21.270705 29729 raft_consensus.cc:1203] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:01:21.339659 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:21.345198 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:21.353420 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:21.362092 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:21.363821 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:21.380561 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:21.403087 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:21.406778 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:21.408658 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:21.487579 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:21.535455 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:21.540547 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:21.543901 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:21.548290 15443 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:21.548406 15443 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:21.548883 15443 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:21.549415 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:21.549548 32255 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
W20240502 14:01:21.549978 29799 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:21.550738 29797 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:21.550806 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:21.551040 15443 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:21.554378 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:21.567456 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:21.576679 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:21.581336 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:21.586954 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:21.588160 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:21.600134 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:21.605283 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:21.670061 29600 consensus_queue.cc:237] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:21.671352 29864 raft_consensus.cc:1261] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:21.671440 29469 raft_consensus.cc:1261] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer d6e2e2c10def4331bec17bd110c9b423: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:21.671794 13896 consensus_queue.cc:1035] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:21.672055 13722 consensus_queue.cc:1035] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:21.674054 13896 raft_consensus.cc:2932] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:21.674651 29469 raft_consensus.cc:2932] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:21.675782 29864 raft_consensus.cc:2932] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:21.677836 29328 catalog_manager.cc:5484] T f28e9f5068aa48b5a87841bdf720d9e0 P d6e2e2c10def4331bec17bd110c9b423 reported cstate change: config changed from index 2 to 3, VOTER efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131) evicted. New cstate: current_term: 1 leader_uuid: "d6e2e2c10def4331bec17bd110c9b423" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:21.682304 29713 tablet_service.cc:1514] Processing DeleteTablet for tablet f28e9f5068aa48b5a87841bdf720d9e0 with delete_type TABLET_DATA_TOMBSTONED (TS efdbf7bdfe0b41bdadb1add053b09a7e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54432
I20240502 14:01:21.682657 15565 tablet_replica.cc:330] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: stopping tablet replica
I20240502 14:01:21.682806 15565 raft_consensus.cc:2229] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:21.682976 15565 raft_consensus.cc:2258] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:21.683398 15565 ts_tablet_manager.cc:1875] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:21.685724 15565 ts_tablet_manager.cc:1888] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:21.685806 15565 log.cc:1196] T f28e9f5068aa48b5a87841bdf720d9e0 P efdbf7bdfe0b41bdadb1add053b09a7e: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-2/wal/wals/f28e9f5068aa48b5a87841bdf720d9e0
I20240502 14:01:21.686507 29314 catalog_manager.cc:4830] TS efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105): tablet f28e9f5068aa48b5a87841bdf720d9e0 (table rebalancer_extra_table_56 [id=b5391956f6c34c9d9c1aac54cdbeaa7c]) successfully deleted
W20240502 14:01:21.690765 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:21.695164 32255 raft_consensus.cc:1261] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:21.695581 15443 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:01:21.705385 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:21.749390 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:21.766001 31990 raft_consensus.cc:1261] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:21.766422 15443 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:21.775385 31861 raft_consensus.cc:1261] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:21.776746 15314 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:01:21.791117 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:21.806890 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:21.814693 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:21.847584 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:21.848069 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:21.856487 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:21.856487 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:21.863049 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:21.879757 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:21.895587 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:21.902891 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:21.936870 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:21.949283 29732 consensus_queue.cc:237] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:21.950810 29600 raft_consensus.cc:1261] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:21.950896 29469 raft_consensus.cc:1261] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:21.951784 15129 consensus_queue.cc:1035] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:21.952185 15090 consensus_queue.cc:1035] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:21.954479 15090 raft_consensus.cc:2932] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:21.956627 29471 raft_consensus.cc:2932] T 25d043c246e94f3090295583133ca0cd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:21.959023 29600 raft_consensus.cc:2932] T 25d043c246e94f3090295583133ca0cd P d6e2e2c10def4331bec17bd110c9b423 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:21.959620 29324 catalog_manager.cc:5484] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 2 to 3, VOTER 8973f2527acb48f5ad36589652de73e1 (127.22.55.136) evicted. New cstate: current_term: 1 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:21.962584 32233 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d043c246e94f3090295583133ca0cd with delete_type TABLET_DATA_TOMBSTONED (TS 8973f2527acb48f5ad36589652de73e1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47700
I20240502 14:01:21.964058 15643 tablet_replica.cc:330] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: stopping tablet replica
I20240502 14:01:21.964212 15643 raft_consensus.cc:2229] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:21.964337 15643 raft_consensus.cc:2258] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:21.967283 15643 ts_tablet_manager.cc:1875] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:21.969519 15643 ts_tablet_manager.cc:1888] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:21.969585 15643 log.cc:1196] T 25d043c246e94f3090295583133ca0cd P 8973f2527acb48f5ad36589652de73e1: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-7/wal/wals/25d043c246e94f3090295583133ca0cd
I20240502 14:01:21.970350 29315 catalog_manager.cc:4830] TS 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201): tablet 25d043c246e94f3090295583133ca0cd (table rebalancer_extra_table_26 [id=c250dd8ffe654dcdb21d816d37df8919]) successfully deleted
W20240502 14:01:21.980966 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:21.985327 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:21.990446 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.015779 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.063407 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:22.124873 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:22.128331 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:22.129137 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:22.139509 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:22.143704 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:22.143913 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:22.157397 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:22.162115 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:22.179208 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:22.193974 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.226824 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.300565 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.308969 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:22.357604 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:22.364347 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:22.368552 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:22.369765 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:22.373620 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:22.374455 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:22.379164 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:22.381830 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.401412 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.409386 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:22.419066 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:22.434317 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.461314 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.468495 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.483486 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:22.486807 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.533787 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.576170 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:22.584543 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:22.587289 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:22.589891 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:22.596940 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:22.601763 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:22.604287 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:22.608213 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:22.626878 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:22.640937 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.641500 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.650471 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.657605 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.659543 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.698297 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:22.716611 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.787580 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.790293 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:22.803177 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:22.804401 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:22.811517 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:22.820953 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:22.825379 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:22.825644 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:22.831014 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:22.852525 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.856899 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.858160 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.892462 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:22.907032 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.965587 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:22.981285 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:23.021199 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:23.023844 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:23.027546 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:23.028635 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:23.033608 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:23.038883 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:23.045887 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:23.046149 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:23.101562 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:23.104691 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.152900 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240502 14:01:23.155684 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.189043 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.190308 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:01:23.191290 31794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:01:23.192059 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.240360 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 14:01:23.244966 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:23.245113 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:23.246961 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:23.247982 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:23.249404 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:23.256974 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:23.267063 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:23.341902 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:01:23.401700 29535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:01:23.444952 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:23.452534 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:23.458026 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:23.462985 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:23.467609 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:23.467679 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:23.473496 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:23.475886 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:23.484695 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 14:01:23.487531 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:23.504228 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.551743 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:23.566469 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:23.568630 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.579135 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:23.665853 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:23.668417 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:23.669302 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:23.673499 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:23.677484 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:23.711350 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:23.712460 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:23.778254 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.874917 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:23.879714 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:23.880805 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:23.881281 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:23.889303 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:23.891822 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:23.898758 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:23.907193 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:23.921811 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:23.949271 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.030341 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.036055 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:24.045480 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:24.051332 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.052740 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:24.101775 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:24.105046 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:24.109529 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:24.112406 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:24.123210 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:24.126329 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:24.148689 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:24.154423 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:24.216913 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:24.242367 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:24.309302 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:24.311543 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:24.315546 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:24.318938 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:24.326288 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:24.332939 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:24.346596 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:24.389766 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.477434 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.495298 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.514197 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.515223 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.562250 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:24.562793 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.602504 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.607225 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:24.616434 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:24.665518 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.719297 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:24.731206 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:24.731787 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:24.732223 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:24.744731 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:24.746716 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:24.752898 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:24.753436 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:24.778982 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:24.789736 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.830447 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:24.838416 32254 consensus_queue.cc:237] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:24.839574 29732 raft_consensus.cc:1261] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:24.839643 29864 raft_consensus.cc:1261] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:24.840191 15135 consensus_queue.cc:1035] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:24.840346 15120 consensus_queue.cc:1035] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:01:24.842901 15120 raft_consensus.cc:2932] T ddb52469e62d4e6ab5bcc90cea577c9c P 8973f2527acb48f5ad36589652de73e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:24.843318 29864 raft_consensus.cc:2932] T ddb52469e62d4e6ab5bcc90cea577c9c P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:24.843732 29732 raft_consensus.cc:2932] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:24.846776 29325 catalog_manager.cc:5484] T ddb52469e62d4e6ab5bcc90cea577c9c P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: config changed from index 2 to 3, VOTER 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129) evicted. New cstate: current_term: 1 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:24.851763 29451 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb52469e62d4e6ab5bcc90cea577c9c with delete_type TABLET_DATA_TOMBSTONED (TS 1d00d3a47c52478894cf8c66959ef7bd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57986
I20240502 14:01:24.852691 16318 tablet_replica.cc:330] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: stopping tablet replica
I20240502 14:01:24.852871 16318 raft_consensus.cc:2229] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:24.853045 16318 raft_consensus.cc:2258] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:24.854467 16318 ts_tablet_manager.cc:1875] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:24.857295 16318 ts_tablet_manager.cc:1888] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 14:01:24.857424 16318 log.cc:1196] T ddb52469e62d4e6ab5bcc90cea577c9c P 1d00d3a47c52478894cf8c66959ef7bd: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-0/wal/wals/ddb52469e62d4e6ab5bcc90cea577c9c
I20240502 14:01:24.858121 29315 catalog_manager.cc:4830] TS 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): tablet ddb52469e62d4e6ab5bcc90cea577c9c (table rebalancer_extra_table_44 [id=6b84752b17254ea3851dfed4e6fb1d04]) successfully deleted
W20240502 14:01:24.864847 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:24.890924 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:24.912617 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:24.937942 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:24.966066 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:24.989717 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:24.999192 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:25.000394 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:25.001961 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:25.009531 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:25.011412 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:25.015578 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:25.017243 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:25.019256 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:25.031044 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:25.040431 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.061538 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:25.077121 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.152834 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.192195 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.193456 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.215916 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.216967 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:25.219621 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:25.219844 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:25.225567 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:25.231029 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:25.231093 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:25.234513 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:25.234751 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:25.236213 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:25.255331 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:25.278975 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.293246 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.340219 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.341214 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:25.362270 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.393939 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.404183 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.415448 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:25.418303 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:25.420578 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:25.433835 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:25.434298 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:25.438755 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:25.438978 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:25.453717 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:25.465445 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:25.489953 16028 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:25.490087 16028 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:25.490598 16028 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:25.490800 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:25.490965 32254 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:25.491588 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:25.492511 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:25.492614 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:25.492926 16028 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:25.493060 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:25.524333 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:25.626147 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:25.628684 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:25.630650 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 14:01:25.638747 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:25.639634 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:25.641800 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:25.642407 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:25.658475 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:25.664418 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:25.731063 29864 consensus_queue.cc:237] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:25.732260 31992 raft_consensus.cc:1261] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:25.732436 32254 raft_consensus.cc:1261] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:25.732789 15973 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:25.733136 15605 consensus_queue.cc:1035] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:25.735039 15973 raft_consensus.cc:2932] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:25.737047 32255 raft_consensus.cc:2932] T ea104c9432a7416f87760e7f6458d6cb P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:25.739185 29322 catalog_manager.cc:5484] T ea104c9432a7416f87760e7f6458d6cb P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: config changed from index 2 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:25.743865 31840 tablet_service.cc:1514] Processing DeleteTablet for tablet ea104c9432a7416f87760e7f6458d6cb with delete_type TABLET_DATA_TOMBSTONED (TS f0e01c4f581843e9ad4ced97f0072dc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43670
I20240502 14:01:25.744174 16547 tablet_replica.cc:330] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: stopping tablet replica
I20240502 14:01:25.744319 16547 raft_consensus.cc:2229] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:25.744446 16547 raft_consensus.cc:2258] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:25.744851 16547 ts_tablet_manager.cc:1875] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:25.748589 16547 ts_tablet_manager.cc:1888] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:01:25.748663 16547 log.cc:1196] T ea104c9432a7416f87760e7f6458d6cb P f0e01c4f581843e9ad4ced97f0072dc4: Deleting WAL directory at /tmp/dist-test-taskCplTwR/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714658376276470-22750-0/raft_consensus-itest-cluster/ts-4/wal/wals/ea104c9432a7416f87760e7f6458d6cb
I20240502 14:01:25.749428 29313 catalog_manager.cc:4830] TS f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): tablet ea104c9432a7416f87760e7f6458d6cb (table rebalancer_extra_table_32 [id=97dffec0264842c2b0959f8f09d91144]) successfully deleted
I20240502 14:01:25.750459 31992 raft_consensus.cc:2932] T ea104c9432a7416f87760e7f6458d6cb P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:01:25.775460 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.789137 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.850150 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:25.889468 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:25.892163 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:25.895146 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:25.898439 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:25.901866 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:25.904963 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:25.916210 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:25.953058 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:25.971808 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:26.001207 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:26.011859 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:26.049438 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:26.112313 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:26.116276 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:26.118467 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:26.121970 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.131058 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:26.131435 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:26.131625 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:26.202298 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:26.235605 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:26.309367 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:26.310266 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.310707 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:26.324352 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:26.328562 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:26.332955 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:26.334394 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:26.336580 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:26.367252 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:26.447985 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:26.473888 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:26.483179 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:26.506330 16508 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:26.506467 16508 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:26.506973 16508 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:26.507247 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:26.507371 29863 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:26.507668 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:26.509140 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:26.509239 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:26.509497 16508 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:26.533586 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:26.534932 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:26.549029 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.553100 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:26.554366 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.559305 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:26.565034 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:26.576495 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:26.578305 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:26.581130 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:26.631098 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:26.727164 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:26.759632 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:26.767470 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:26.774142 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:26.775336 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.779331 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:26.779729 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.785045 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:26.814043 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:26.895478 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:26.908339 31794 consensus_peers.cc:485] T 09f77f3b6bf14a46b2e41c91125f6bd7 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:26.970957 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.978142 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:26.979384 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:26.981853 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:26.982262 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:26.982786 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:26.983937 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:27.044509 29535 consensus_peers.cc:485] T 253ea421888249d180928ba0aef28ddd P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.046380 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.051187 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:27.090938 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:27.130148 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.154836 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.159672 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.172945 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.191411 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:27.209059 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:27.209425 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:27.211597 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:27.212579 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:27.220001 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:27.227902 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:27.231335 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:27.252516 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:27.261135 29535 consensus_peers.cc:485] T a63de5b3f35e4486859c9361925689d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.271684 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.310869 31794 consensus_peers.cc:485] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.335811 29535 consensus_peers.cc:485] T a50a38153ff44d1c988838bb39884f20 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.343281 31794 consensus_peers.cc:485] T fbdb9b4a6c07488f9bc2d15cba59214e P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:27.419391 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:27.422045 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:27.429003 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:27.430670 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:27.431324 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:27.433182 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:27.442610 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:27.454591 16028 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:27.454691 16028 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:27.455112 16028 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
W20240502 14:01:27.455749 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:27.461172 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:27.461237 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:27.461477 16028 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:27.462906 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:27.463037 32255 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:27.466568 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.471601 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.489035 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.498927 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.504227 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.558404 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.559579 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:27.560163 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.561393 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.597621 31794 consensus_peers.cc:485] T eab1144457d34c25897c5b7a691409fe P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.625291 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.625911 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:27.635320 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:27.646656 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:27.652922 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:27.654306 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:27.657447 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240502 14:01:27.657788 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:27.672331 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:27.682897 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:27.689348 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:27.691581 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 14:01:27.733839 31794 consensus_peers.cc:485] T fe523f0c364f447cb2a62b599d030cca P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.752985 29535 consensus_peers.cc:485] T 0867c6a22c2c4efd9a9babff371cc8ba P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.788254 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.798187 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.811754 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:27.853120 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.865212 29535 consensus_peers.cc:485] T bd91ed1ae1354a84a0ff668efdbc03d9 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:27.872939 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:27.897517 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:27.897671 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:27.897681 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:27.903650 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:27.905889 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:27.906541 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240502 14:01:27.906720 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:27.931015 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:27.973197 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.043565 31794 consensus_peers.cc:485] T 8579d75d058e463b85fa5c982d74bf3f P f0e01c4f581843e9ad4ced97f0072dc4 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.066279 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:28.120472 29843 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:28.123143 29451 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:01:28.129038 31970 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:01:28.130693 29582 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 14:01:28.134277 29535 consensus_peers.cc:485] T 75347e9c867547efaf78355aec6dfb40 P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:28.147764 31840 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:01:28.151923 32233 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240502 14:01:28.155797 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:28.156325 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 160 similar messages]
I20240502 14:01:28.164316 29713 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:01:28.214290 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:01:28.223013 29471 consensus_queue.cc:237] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:28.224604 29863 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:28.224615 31990 raft_consensus.cc:1261] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:01:28.224808 29404 consensus_peers.cc:485] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f1a1a1592e84b3898a06d4360bac846. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:28.225020 17140 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:28.225195 16028 consensus_queue.cc:1035] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:28.227701 17019 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:28.229616 29863 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:28.230320 31990 raft_consensus.cc:2932] T 6f1a1a1592e84b3898a06d4360bac846 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:28.231490 29327 catalog_manager.cc:5484] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: config changed from index 5 to 6, VOTER f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) added. New cstate: current_term: 1 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
I20240502 14:01:28.253957 31990 consensus_queue.cc:237] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:28.255694 29732 raft_consensus.cc:1261] T 65229e977b8543639fb27bbbc3b5e797 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:01:28.256318 16965 consensus_queue.cc:1035] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:01:28.256453 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65229e977b8543639fb27bbbc3b5e797. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.256810 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.257448 31794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:01:28.269623 29535 consensus_peers.cc:485] T 21a2c1b2947c47289ace8cf41f2fbb4d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.285167 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.335508 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.344810 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:01:28.349357 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 14:01:28.432886 29535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:01:28.450747 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.498412 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.527407 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 14:01:28.569550 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.603924 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.653949 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:28.660854 29535 consensus_peers.cc:485] T 6d3a84c1d004412db381854147330dfc P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:28.672919 17166 ts_tablet_manager.cc:902] T 6f1a1a1592e84b3898a06d4360bac846 P f0e01c4f581843e9ad4ced97f0072dc4: Initiating tablet copy from peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
F20240502 14:01:28.673192 17166 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1089cd67cbd24551836ccb702ff44d4a.queue_time_us, tablet-download-pool-1089cd67cbd24551836ccb702ff44d4a.queue_time_us) (tablet-download-pool-1089cd67cbd24551836ccb702ff44d4a.run_cpu_time_us, tablet-download-pool-1089cd67cbd24551836ccb702ff44d4a.run_cpu_time_us) (tablet-download-pool-1089cd67cbd24551836ccb702ff44d4a.run_wall_time_us, tablet-download-pool-1089cd67cbd24551836ccb702ff44d4a.run_wall_time_us) (tablet-download-pool-1fc14b4bd42841df91972e0c5d02faa4.queue_time_us, tablet-download-pool-1fc14b4bd42841df91972e0c5d02faa4.queue_time_us) (tablet-download-pool-1fc14b4bd42841df91972e0c5d02faa4.run_cpu_time_us, tablet-download-pool-1fc14b4bd42841df91972e0c5d02faa4.run_cpu_time_us) (tablet-download-pool-1fc14b4bd42841df91972e0c5d02faa4.run_wall_time_us, tablet-download-pool-1fc14b4bd42841df91972e0c5d02faa4.run_wall_time_us) (tablet-download-pool-209e6922777b49ddb87a3dadf65d651a.queue_time_us, tablet-download-pool-209e6922777b49ddb87a3dadf65d651a.queue_time_us) (tablet-download-pool-209e6922777b49ddb87a3dadf65d651a.run_cpu_time_us, tablet-download-pool-209e6922777b49ddb87a3dadf65d651a.run_cpu_time_us) (tablet-download-pool-209e6922777b49ddb87a3dadf65d651a.run_wall_time_us, tablet-download-pool-209e6922777b49ddb87a3dadf65d651a.run_wall_time_us) (tablet-download-pool-253ea421888249d180928ba0aef28ddd.queue_time_us, tablet-download-pool-253ea421888249d180928ba0aef28ddd.queue_time_us) (tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_cpu_time_us, tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_cpu_time_us) (tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_wall_time_us, tablet-download-pool-253ea421888249d180928ba0aef28ddd.run_wall_time_us) (tablet-download-pool-463dddd3473a42a68b488e1c465e997f.queue_time_us, tablet-download-pool-463dddd3473a42a68b488e1c465e997f.queue_time_us) (tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_cpu_time_us, tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_cpu_time_us) (tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_wall_time_us, tablet-download-pool-463dddd3473a42a68b488e1c465e997f.run_wall_time_us) (tablet-download-pool-59235c5f3daf4ff1afeab83efe018405.queue_time_us, tablet-download-pool-59235c5f3daf4ff1afeab83efe018405.queue_time_us) (tablet-download-pool-59235c5f3daf4ff1afeab83efe018405.run_cpu_time_us, tablet-download-pool-59235c5f3daf4ff1afeab83efe018405.run_cpu_time_us) (tablet-download-pool-59235c5f3daf4ff1afeab83efe018405.run_wall_time_us, tablet-download-pool-59235c5f3daf4ff1afeab83efe018405.run_wall_time_us) (tablet-download-pool-6767168f6f2b4e15aab7e64b0eab57c3.queue_time_us, tablet-download-pool-6767168f6f2b4e15aab7e64b0eab57c3.queue_time_us) (tablet-download-pool-6767168f6f2b4e15aab7e64b0eab57c3.run_cpu_time_us, tablet-download-pool-6767168f6f2b4e15aab7e64b0eab57c3.run_cpu_time_us) (tablet-download-pool-6767168f6f2b4e15aab7e64b0eab57c3.run_wall_time_us, tablet-download-pool-6767168f6f2b4e15aab7e64b0eab57c3.run_wall_time_us) (tablet-download-pool-6eae206e12b44b809d8b2fcc21611a28.queue_time_us, tablet-download-pool-6eae206e12b44b809d8b2fcc21611a28.queue_time_us) (tablet-download-pool-6eae206e12b44b809d8b2fcc21611a28.run_cpu_time_us, tablet-download-pool-6eae206e12b44b809d8b2fcc21611a28.run_cpu_time_us) (tablet-download-pool-6eae206e12b44b809d8b2fcc21611a28.run_wall_time_us, tablet-download-pool-6eae206e12b44b809d8b2fcc21611a28.run_wall_time_us) (tablet-download-pool-6f1a1a1592e84b3898a06d4360bac846.queue_time_us, tablet-download-pool-6f1a1a1592e84b3898a06d4360bac846.queue_time_us) (tablet-download-pool-7650009dd8954f11ab2ecafef765692c.queue_time_us, tablet-download-pool-7650009dd8954f11ab2ecafef765692c.queue_time_us) (tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_cpu_time_us, tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_cpu_time_us) (tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_wall_time_us, tablet-download-pool-7650009dd8954f11ab2ecafef765692c.run_wall_time_us) (tablet-download-pool-8d2ab1ded2c241038d5685678aa819a8.queue_time_us, tablet-download-pool-8d2ab1ded2c241038d5685678aa819a8.queue_time_us) (tablet-download-pool-8d2ab1ded2c241038d5685678aa819a8.run_cpu_time_us, tablet-download-pool-8d2ab1ded2c241038d5685678aa819a8.run_cpu_time_us) (tablet-download-pool-8d2ab1ded2c241038d5685678aa819a8.run_wall_time_us, tablet-download-pool-8d2ab1ded2c241038d5685678aa819a8.run_wall_time_us) (tablet-download-pool-995449d84e4b4d4692f8f225b5a44830.queue_time_us, tablet-download-pool-995449d84e4b4d4692f8f225b5a44830.queue_time_us) (tablet-download-pool-995449d84e4b4d4692f8f225b5a44830.run_cpu_time_us, tablet-download-pool-995449d84e4b4d4692f8f225b5a44830.run_cpu_time_us) (tablet-download-pool-995449d84e4b4d4692f8f225b5a44830.run_wall_time_us, tablet-download-pool-995449d84e4b4d4692f8f225b5a44830.run_wall_time_us) (tablet-download-pool-9bdf925a9c4d456fb3b4ef448a2154c6.queue_time_us, tablet-download-pool-9bdf925a9c4d456fb3b4ef448a2154c6.queue_time_us) (tablet-download-pool-9bdf925a9c4d456fb3b4ef448a2154c6.run_cpu_time_us, tablet-download-pool-9bdf925a9c4d456fb3b4ef448a2154c6.run_cpu_time_us) (tablet-download-pool-9bdf925a9c4d456fb3b4ef448a2154c6.run_wall_time_us, tablet-download-pool-9bdf925a9c4d456fb3b4ef448a2154c6.run_wall_time_us) (tablet-download-pool-aeaf1db0298741fa813660a8e38d01f9.queue_time_us, tablet-download-pool-aeaf1db0298741fa813660a8e38d01f9.queue_time_us) (tablet-download-pool-aeaf1db0298741fa813660a8e38d01f9.run_cpu_time_us, tablet-download-pool-aeaf1db0298741fa813660a8e38d01f9.run_cpu_time_us) (tablet-download-pool-aeaf1db0298741fa813660a8e38d01f9.run_wall_time_us, tablet-download-pool-aeaf1db0298741fa813660a8e38d01f9.run_wall_time_us) (tablet-download-pool-c35dda6845eb484886eed36e40b9231c.queue_time_us, tablet-download-pool-c35dda6845eb484886eed36e40b9231c.queue_time_us) (tablet-download-pool-c35dda6845eb484886eed36e40b9231c.run_cpu_time_us, tablet-download-pool-c35dda6845eb484886eed36e40b9231c.run_cpu_time_us) (tablet-download-pool-c35dda6845eb484886eed36e40b9231c.run_wall_time_us, tablet-download-pool-c35dda6845eb484886eed36e40b9231c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f4e9408cdcd google::LogMessage::Fail() at ??:0
@ 0x7f4e94090b93 google::LogMessage::SendToLog() at ??:0
W20240502 14:01:28.677985 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
@ 0x7f4e9408c7cc google::LogMessage::Flush() at ??:0
@ 0x7f4e9408df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4e954f6044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f4e954daf98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4e954d8efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f4e9808b77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f4e98094a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f4e9811576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240502 14:01:28.701460 29535 consensus_peers.cc:462] T 7ece4fc38b254afeae58936444fe047d P d6e2e2c10def4331bec17bd110c9b423 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
@ 0x7f4e981294f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f4e98114226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f4e98123c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ebd8e6b75c std::function<>::operator()() at ??:0
@ 0x7f4e954dde06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e954de6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4e954e003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ebd8e6b75c std::function<>::operator()() at ??:0
@ 0x7f4e954cd807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e95d0b6db start_thread at ??:0
@ 0x7f4e9324371f clone at ??:0
I20240502 14:01:28.825505 17188 ts_tablet_manager.cc:902] T 65229e977b8543639fb27bbbc3b5e797 P d6e2e2c10def4331bec17bd110c9b423: Initiating tablet copy from peer 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
F20240502 14:01:28.825696 17188 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-04f0f9efcbb544d7b8292158bf769426.queue_time_us, tablet-download-pool-04f0f9efcbb544d7b8292158bf769426.queue_time_us) (tablet-download-pool-04f0f9efcbb544d7b8292158bf769426.run_cpu_time_us, tablet-download-pool-04f0f9efcbb544d7b8292158bf769426.run_cpu_time_us) (tablet-download-pool-04f0f9efcbb544d7b8292158bf769426.run_wall_time_us, tablet-download-pool-04f0f9efcbb544d7b8292158bf769426.run_wall_time_us) (tablet-download-pool-09f77f3b6bf14a46b2e41c91125f6bd7.queue_time_us, tablet-download-pool-09f77f3b6bf14a46b2e41c91125f6bd7.queue_time_us) (tablet-download-pool-09f77f3b6bf14a46b2e41c91125f6bd7.run_cpu_time_us, tablet-download-pool-09f77f3b6bf14a46b2e41c91125f6bd7.run_cpu_time_us) (tablet-download-pool-09f77f3b6bf14a46b2e41c91125f6bd7.run_wall_time_us, tablet-download-pool-09f77f3b6bf14a46b2e41c91125f6bd7.run_wall_time_us) (tablet-download-pool-19d48f827a6243448d344ee44a54482d.queue_time_us, tablet-download-pool-19d48f827a6243448d344ee44a54482d.queue_time_us) (tablet-download-pool-19d48f827a6243448d344ee44a54482d.run_cpu_time_us, tablet-download-pool-19d48f827a6243448d344ee44a54482d.run_cpu_time_us) (tablet-download-pool-19d48f827a6243448d344ee44a54482d.run_wall_time_us, tablet-download-pool-19d48f827a6243448d344ee44a54482d.run_wall_time_us) (tablet-download-pool-1b37defc84d34af08f5744bf60a038c5.queue_time_us, tablet-download-pool-1b37defc84d34af08f5744bf60a038c5.queue_time_us) (tablet-download-pool-1b37defc84d34af08f5744bf60a038c5.run_cpu_time_us, tablet-download-pool-1b37defc84d34af08f5744bf60a038c5.run_cpu_time_us) (tablet-download-pool-1b37defc84d34af08f5744bf60a038c5.run_wall_time_us, tablet-download-pool-1b37defc84d34af08f5744bf60a038c5.run_wall_time_us) (tablet-download-pool-1ea3465ad91540cea8906bcc406efd07.queue_time_us, tablet-download-pool-1ea3465ad91540cea8906bcc406efd07.queue_time_us) (tablet-download-pool-1ea3465ad91540cea8906bcc406efd07.run_cpu_time_us, tablet-download-pool-1ea3465ad91540cea8906bcc406efd07.run_cpu_time_us) (tablet-download-pool-1ea3465ad91540cea8906bcc406efd07.run_wall_time_us, tablet-download-pool-1ea3465ad91540cea8906bcc406efd07.run_wall_time_us) (tablet-download-pool-25d043c246e94f3090295583133ca0cd.queue_time_us, tablet-download-pool-25d043c246e94f3090295583133ca0cd.queue_time_us) (tablet-download-pool-25d043c246e94f3090295583133ca0cd.run_cpu_time_us, tablet-download-pool-25d043c246e94f3090295583133ca0cd.run_cpu_time_us) (tablet-download-pool-25d043c246e94f3090295583133ca0cd.run_wall_time_us, tablet-download-pool-25d043c246e94f3090295583133ca0cd.run_wall_time_us) (tablet-download-pool-3062c7a80b284f6c8cb28f8c53c9c0a4.queue_time_us, tablet-download-pool-3062c7a80b284f6c8cb28f8c53c9c0a4.queue_time_us) (tablet-download-pool-3062c7a80b284f6c8cb28f8c53c9c0a4.run_cpu_time_us, tablet-download-pool-3062c7a80b284f6c8cb28f8c53c9c0a4.run_cpu_time_us) (tablet-download-pool-3062c7a80b284f6c8cb28f8c53c9c0a4.run_wall_time_us, tablet-download-pool-3062c7a80b284f6c8cb28f8c53c9c0a4.run_wall_time_us) (tablet-download-pool-541806c11f074316a5342050427f4849.queue_time_us, tablet-download-pool-541806c11f074316a5342050427f4849.queue_time_us) (tablet-download-pool-541806c11f074316a5342050427f4849.run_cpu_time_us, tablet-download-pool-541806c11f074316a5342050427f4849.run_cpu_time_us) (tablet-download-pool-541806c11f074316a5342050427f4849.run_wall_time_us, tablet-download-pool-541806c11f074316a5342050427f4849.run_wall_time_us) (tablet-download-pool-65229e977b8543639fb27bbbc3b5e797.queue_time_us, tablet-download-pool-65229e977b8543639fb27bbbc3b5e797.queue_time_us) (tablet-download-pool-6d3a84c1d004412db381854147330dfc.queue_time_us, tablet-download-pool-6d3a84c1d004412db381854147330dfc.queue_time_us) (tablet-download-pool-6d3a84c1d004412db381854147330dfc.run_cpu_time_us, tablet-download-pool-6d3a84c1d004412db381854147330dfc.run_cpu_time_us) (tablet-download-pool-6d3a84c1d004412db381854147330dfc.run_wall_time_us, tablet-download-pool-6d3a84c1d004412db381854147330dfc.run_wall_time_us) (tablet-download-pool-75347e9c867547efaf78355aec6dfb40.queue_time_us, tablet-download-pool-75347e9c867547efaf78355aec6dfb40.queue_time_us) (tablet-download-pool-75347e9c867547efaf78355aec6dfb40.run_cpu_time_us, tablet-download-pool-75347e9c867547efaf78355aec6dfb40.run_cpu_time_us) (tablet-download-pool-75347e9c867547efaf78355aec6dfb40.run_wall_time_us, tablet-download-pool-75347e9c867547efaf78355aec6dfb40.run_wall_time_us) (tablet-download-pool-768df43846d94951a3a48f6f626bb473.queue_time_us, tablet-download-pool-768df43846d94951a3a48f6f626bb473.queue_time_us) (tablet-download-pool-768df43846d94951a3a48f6f626bb473.run_cpu_time_us, tablet-download-pool-768df43846d94951a3a48f6f626bb473.run_cpu_time_us) (tablet-download-pool-768df43846d94951a3a48f6f626bb473.run_wall_time_us, tablet-download-pool-768df43846d94951a3a48f6f626bb473.run_wall_time_us) (tablet-download-pool-8eb9e9de44a046ffb4594755f44f15b8.queue_time_us, tablet-download-pool-8eb9e9de44a046ffb4594755f44f15b8.queue_time_us) (tablet-download-pool-8eb9e9de44a046ffb4594755f44f15b8.run_cpu_time_us, tablet-download-pool-8eb9e9de44a046ffb4594755f44f15b8.run_cpu_time_us) (tablet-download-pool-8eb9e9de44a046ffb4594755f44f15b8.run_wall_time_us, tablet-download-pool-8eb9e9de44a046ffb4594755f44f15b8.run_wall_time_us) (tablet-download-pool-a0001d3f124c42eba6129836209a41fa.queue_time_us, tablet-download-pool-a0001d3f124c42eba6129836209a41fa.queue_time_us) (tablet-download-pool-a0001d3f124c42eba6129836209a41fa.run_cpu_time_us, tablet-download-pool-a0001d3f124c42eba6129836209a41fa.run_cpu_time_us) (tablet-download-pool-a0001d3f124c42eba6129836209a41fa.run_wall_time_us, tablet-download-pool-a0001d3f124c42eba6129836209a41fa.run_wall_time_us) (tablet-download-pool-b0fa578318964569b2e10d7f858acd4d.queue_time_us, tablet-download-pool-b0fa578318964569b2e10d7f858acd4d.queue_time_us) (tablet-download-pool-b0fa578318964569b2e10d7f858acd4d.run_cpu_time_us, tablet-download-pool-b0fa578318964569b2e10d7f858acd4d.run_cpu_time_us) (tablet-download-pool-b0fa578318964569b2e10d7f858acd4d.run_wall_time_us, tablet-download-pool-b0fa578318964569b2e10d7f858acd4d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f664ebabdcd google::LogMessage::Fail() at ??:0
@ 0x7f664ebafb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f664ebab7cc google::LogMessage::Flush() at ??:0
@ 0x7f664ebacf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6650015044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f664fff9f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f664fff7efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f6652baa77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f6652bb3a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f6652c3476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f6652c484f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f6652c33226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f6652c42c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55b5d48bd75c std::function<>::operator()() at ??:0
@ 0x7f664fffce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f664fffd6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f664ffff03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55b5d48bd75c std::function<>::operator()() at ??:0
@ 0x7f664ffec807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f665082a6db start_thread at ??:0
@ 0x7f664dd6271f clone at ??:0
W20240502 14:01:28.931856 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:28.954492 29797 connection.cc:535] client connection to 127.22.55.133:39647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:28.954553 31925 connection.cc:535] client connection to 127.22.55.133:39647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:28.954965 29666 connection.cc:535] client connection to 127.22.55.133:39647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:28.955212 29404 connection.cc:535] client connection to 127.22.55.133:39647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:28.955652 32188 connection.cc:535] client connection to 127.22.55.133:39647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:28.956537 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:28.955036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.133:39647 (local address 127.22.55.134:47437)
0502 14:01:28.955145 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:28.955159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:28.955259 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:28.956398 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
W20240502 14:01:28.956580 11585 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:28.956126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.133:39647 (local address 127.22.55.136:41159)
0502 14:01:28.956269 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:28.956283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:28.956388 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:28.956478 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52}
W20240502 14:01:28.956660 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.956730 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.956838 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.957373 11587 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:28.955841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.133:39647 (local address 127.22.55.129:38809)
0502 14:01:28.956085 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:28.956097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:28.956178 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:28.956808 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":167}
W20240502 14:01:28.957526 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.957592 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.957640 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:28.957635 11578 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:28.955590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.133:39647 (local address 127.22.55.131:33519)
0502 14:01:28.955768 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:28.955781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:28.955861 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:28.957185 (+ 1324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":102}
W20240502 14:01:28.958022 11584 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:28.955112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.133:39647 (local address 127.22.55.132:38993)
0502 14:01:28.955201 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:28.955215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:28.955321 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:28.957932 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20240502 14:01:28.958194 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.958786 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.958868 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.958928 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.966315 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.970933 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.988318 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:28.992841 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.008886 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:29.009456 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.026696 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.050968 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.061717 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.064432 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.064735 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.066452 31925 connection.cc:535] client connection to 127.22.55.130:40041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:29.066593 29666 connection.cc:535] client connection to 127.22.55.130:40041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:29.066879 32189 connection.cc:535] server connection from 127.22.55.130:39449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:29.066980 32188 connection.cc:535] client connection to 127.22.55.130:40041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:29.068078 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:29.068259 29797 connection.cc:535] client connection to 127.22.55.130:40041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:29.068293 29404 connection.cc:535] client connection to 127.22.55.130:40041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:29.069260 11584 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:29.068742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.130:40041 (local address 127.22.55.132:34651)
0502 14:01:29.068856 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:29.068869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:29.068953 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:29.069177 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
W20240502 14:01:29.069260 11578 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:29.067234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.130:40041 (local address 127.22.55.131:53245)
0502 14:01:29.067324 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:29.067338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:29.067415 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:29.069168 (+ 1753us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24}
W20240502 14:01:29.069262 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:29.067158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.130:40041 (local address 127.22.55.134:52845)
0502 14:01:29.067284 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:29.067298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:29.067389 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:29.069168 (+ 1779us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
W20240502 14:01:29.069435 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.069499 11585 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:29.067585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.130:40041 (local address 127.22.55.136:54281)
0502 14:01:29.067682 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:29.067692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:29.067752 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:29.069426 (+ 1674us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":40}
W20240502 14:01:29.069491 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:29.069617 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.069751 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.069757 11587 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:29.068873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.55.130:40041 (local address 127.22.55.129:37815)
0502 14:01:29.068993 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:29.069007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:29.069081 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:29.069658 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":46}
W20240502 14:01:29.069795 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.069875 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.069941 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.069967 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.086439 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.090755 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.105468 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.115813 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.116279 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.126753 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.129012 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.135591 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.149725 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.171334 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.173247 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.174222 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.183050 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.189666 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.197670 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.200630 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.209381 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.213161 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.213526 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.222396 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:29.222473 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.229058 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.232256 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:29.255769 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.259696 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.262941 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.267068 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.275161 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.275403 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.290730 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.329459 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.350337 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.435084 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.456590 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:29.499243 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:29.516067 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.518261 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.539498 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.540963 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.571287 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:29.612542 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:29.629804 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:29.638727 16508 raft_consensus.cc:482] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.638828 16508 raft_consensus.cc:504] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:29.639328 16508 leader_election.cc:288] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:29.639678 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e9d8ef4f754df3817cc4a981549e02" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
W20240502 14:01:29.640471 32188 leader_election.cc:334] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.640545 32188 leader_election.cc:302] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.640668 16508 raft_consensus.cc:2726] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:29.712920 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:29.724115 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:29.728034 15120 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.728137 15120 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:29.728559 15120 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:29.728876 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
W20240502 14:01:29.729840 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.729913 32188 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:29.730232 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.731274 15120 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.765259 17163 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.765369 17163 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:29.765699 17163 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:29.766700 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:29.766824 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.766913 31925 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:29.767226 17163 raft_consensus.cc:2726] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.773687 17019 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.773773 17019 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:29.774096 17019 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:29.775027 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.775131 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.775178 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.775293 17019 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.779647 17163 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.779747 17163 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:29.780073 17163 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:29.781038 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.781150 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.781196 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.781338 17163 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:29.799238 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:29.800448 17163 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.800532 17163 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:29.800913 17163 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:29.801273 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
W20240502 14:01:29.801977 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.802034 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.802182 17163 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:29.802173 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.813977 17019 raft_consensus.cc:482] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.814081 17019 raft_consensus.cc:504] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:29.814424 17019 leader_election.cc:288] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:29.814734 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631725c869674c45ac79c3f05cee8356" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
W20240502 14:01:29.815296 29404 leader_election.cc:334] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.815358 29404 leader_election.cc:302] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.815474 17019 raft_consensus.cc:2726] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.816287 16508 raft_consensus.cc:482] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.816366 16508 raft_consensus.cc:504] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:29.816648 16508 leader_election.cc:288] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:29.817133 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afd58ca519e4bc8a6ab00c37abf9f1a" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
W20240502 14:01:29.817523 32188 leader_election.cc:334] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.817584 32188 leader_election.cc:302] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.817710 16508 raft_consensus.cc:2726] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.830957 17019 raft_consensus.cc:482] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.831045 17019 raft_consensus.cc:504] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:29.831316 17019 leader_election.cc:288] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:29.831656 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2ab852e31d4181a8597b9047a08554" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
W20240502 14:01:29.832342 29404 leader_election.cc:334] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.832404 29404 leader_election.cc:302] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:29.832525 17019 raft_consensus.cc:2726] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.839228 17210 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.839320 17210 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:29.839632 17210 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:29.840615 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:29.840783 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.840842 29797 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:29.840982 17210 raft_consensus.cc:2726] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.843328 17163 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.843415 17163 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:29.843724 17163 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:29.844894 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.844975 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.845018 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:29.845170 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.845166 17163 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:29.860630 17210 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.860618 17019 raft_consensus.cc:482] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.860723 17210 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:29.860723 17019 raft_consensus.cc:504] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:29.861104 17019 leader_election.cc:288] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:29.861101 17210 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:29.861567 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835183257412496b90defa7ecaa0ff13" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:01:29.862053 29404 leader_election.cc:334] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.862124 29404 leader_election.cc:302] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:29.862254 17019 raft_consensus.cc:2726] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:29.862623 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.862740 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.862792 29797 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:29.862943 17210 raft_consensus.cc:2726] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:29.909485 17210 raft_consensus.cc:482] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.909602 17210 raft_consensus.cc:504] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:29.910017 17210 leader_election.cc:288] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:29.910421 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c91754a477542e5abea67a09971d111" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:01:29.910946 29797 leader_election.cc:334] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.911008 29797 leader_election.cc:302] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:29.911123 17210 raft_consensus.cc:2726] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.913260 17019 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.913359 17019 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:29.913831 17019 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:29.914309 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:29.914454 31990 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:29.915021 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.915099 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.915134 29404 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:29.915252 17019 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.918076 16508 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.918190 16508 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:29.918495 16508 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:29.919541 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.919662 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.919714 32188 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.919853 16508 raft_consensus.cc:2726] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.923590 16482 raft_consensus.cc:482] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.923686 16482 raft_consensus.cc:504] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:29.924079 16482 leader_election.cc:288] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:29.924518 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fefaee1ab4c4a77aa7ed79d6c2e8894" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
W20240502 14:01:29.925352 29666 leader_election.cc:334] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.925429 29666 leader_election.cc:302] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.925586 17212 raft_consensus.cc:2726] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.927546 17212 raft_consensus.cc:482] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.927632 17212 raft_consensus.cc:504] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:29.927932 17212 leader_election.cc:288] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:29.928182 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69badccfbc416f92d982cf394ea0cd" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
W20240502 14:01:29.929095 29666 leader_election.cc:334] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.929169 29666 leader_election.cc:302] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:29.929268 17212 raft_consensus.cc:2726] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.943660 16508 raft_consensus.cc:482] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.943763 16508 raft_consensus.cc:504] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:29.944125 16508 leader_election.cc:288] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:29.944432 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835183257412496b90defa7ecaa0ff13" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:29.944566 29471 raft_consensus.cc:2454] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:29.944882 32190 leader_election.cc:302] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:29.945063 16508 raft_consensus.cc:2781] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:29.945117 16508 raft_consensus.cc:482] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.945170 16508 raft_consensus.cc:3037] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:29.945222 32188 leader_election.cc:334] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.945714 17163 raft_consensus.cc:482] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.945817 17163 raft_consensus.cc:504] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:29.946254 17163 leader_election.cc:288] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:29.946254 16508 raft_consensus.cc:504] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:29.946519 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "891851115e2e4d9db5bf3a0013b0edba" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:29.946566 16508 leader_election.cc:288] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
W20240502 14:01:29.947554 31925 leader_election.cc:334] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:29.948388 32188 leader_election.cc:334] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.950318 17163 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.950400 17163 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:29.950735 17163 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:29.951148 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835183257412496b90defa7ecaa0ff13" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:29.951256 29471 raft_consensus.cc:3037] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:29.952122 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.952275 29471 raft_consensus.cc:2454] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
W20240502 14:01:29.952370 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:29.952418 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:29.952517 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.952544 32190 leader_election.cc:302] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:29.952652 17163 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.952782 15120 raft_consensus.cc:2781] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:29.952992 15120 raft_consensus.cc:686] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:29.953138 15120 consensus_queue.cc:237] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:29.955835 29327 catalog_manager.cc:5484] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:01:29.961777 31926 leader_election.cc:302] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.962031 17163 raft_consensus.cc:2726] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:29.963759 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.964136 17163 raft_consensus.cc:482] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.964221 17163 raft_consensus.cc:504] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:29.964620 17163 leader_election.cc:288] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:29.964954 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afd58ca519e4bc8a6ab00c37abf9f1a" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:29.965060 32254 raft_consensus.cc:2454] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:01:29.965322 31927 leader_election.cc:302] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:29.965471 17163 raft_consensus.cc:2781] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:29.965513 17163 raft_consensus.cc:482] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.965548 17163 raft_consensus.cc:3037] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:29.965873 31925 leader_election.cc:334] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.966603 17163 raft_consensus.cc:504] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
I20240502 14:01:29.966962 17163 leader_election.cc:288] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:29.967381 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afd58ca519e4bc8a6ab00c37abf9f1a" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:01:29.967483 32254 raft_consensus.cc:3037] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:29.968446 32254 raft_consensus.cc:2454] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
W20240502 14:01:29.968647 31925 leader_election.cc:334] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.968819 31927 leader_election.cc:302] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 8973f2527acb48f5ad36589652de73e1; no voters: f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.969101 17163 raft_consensus.cc:2781] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:29.969249 17163 raft_consensus.cc:686] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:01:29.969367 17163 consensus_queue.cc:237] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } }
W20240502 14:01:29.969576 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:29.972425 29327 catalog_manager.cc:5484] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } }
I20240502 14:01:29.987459 17224 raft_consensus.cc:482] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.987578 17224 raft_consensus.cc:504] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:29.987949 17224 leader_election.cc:288] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:29.988314 17210 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:29.988303 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c6786a80e40a98130a5d9057e2877" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:29.988421 17210 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:29.988754 17210 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:29.988901 31925 leader_election.cc:334] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.989037 31926 leader_election.cc:302] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: efdbf7bdfe0b41bdadb1add053b09a7e, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.989199 17224 raft_consensus.cc:2726] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:29.989730 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:29.989814 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:29.989881 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:29.990015 17210 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:29.990478 17224 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:29.990574 17224 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:29.990886 17224 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:29.991820 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:29.991935 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:29.991976 31925 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:29.992089 17224 raft_consensus.cc:2726] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.020685 16508 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.020793 16508 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.021173 16508 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.021152 17210 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.021248 17210 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:30.021606 17210 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:30.022183 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:30.022297 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.022351 32188 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.022508 16508 raft_consensus.cc:2726] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:30.022743 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:30.022835 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.022912 29797 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.023032 17210 raft_consensus.cc:2726] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:30.029619 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:30.037387 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:30.038427 17224 raft_consensus.cc:482] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.038518 17224 raft_consensus.cc:504] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.038882 17224 leader_election.cc:288] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:30.039232 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
W20240502 14:01:30.039733 31925 leader_election.cc:334] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.039794 31925 leader_election.cc:302] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.039934 17224 raft_consensus.cc:2726] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.046902 17210 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.047005 17210 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.047344 17210 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:30.048799 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:30.048902 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.048945 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.049062 17210 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.051546 17210 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.051630 17210 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.051915 17210 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.052028 17019 raft_consensus.cc:482] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.052136 17019 raft_consensus.cc:504] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.052515 17019 leader_election.cc:288] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:30.053146 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:30.053272 31992 raft_consensus.cc:2454] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:30.053388 29404 leader_election.cc:334] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.053808 29404 leader_election.cc:302] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.054028 17019 raft_consensus.cc:2781] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.054077 17019 raft_consensus.cc:482] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.054126 17019 raft_consensus.cc:3037] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.054509 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:30.054745 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.054803 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:30.054947 17210 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:30.055351 17019 raft_consensus.cc:504] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.055663 17019 leader_election.cc:288] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:30.056180 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f716493d95f84740beb85c3bf4021ad0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:30.056299 31992 raft_consensus.cc:3037] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.056545 29404 leader_election.cc:334] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.057101 31992 raft_consensus.cc:2454] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 2.
I20240502 14:01:30.057354 29404 leader_election.cc:302] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.057493 17019 raft_consensus.cc:2781] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.057663 17019 raft_consensus.cc:686] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:01:30.057806 17019 consensus_queue.cc:237] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.059935 29327 catalog_manager.cc:5484] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 2 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:01:30.065099 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.074177 17212 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.074291 17212 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.074621 17212 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:30.075621 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:30.075738 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.075809 29666 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:30.075932 17212 raft_consensus.cc:2726] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.083730 17212 raft_consensus.cc:482] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.083832 17212 raft_consensus.cc:504] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:30.084234 17212 leader_election.cc:288] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:30.084607 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "891851115e2e4d9db5bf3a0013b0edba" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:30.084734 31992 raft_consensus.cc:2454] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:01:30.085101 29666 leader_election.cc:302] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
W20240502 14:01:30.085264 29666 leader_election.cc:334] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.085319 17212 raft_consensus.cc:2781] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.085388 17212 raft_consensus.cc:482] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.085433 17212 raft_consensus.cc:3037] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.086632 17212 raft_consensus.cc:504] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:30.086997 17212 leader_election.cc:288] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Requested vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:30.087522 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "891851115e2e4d9db5bf3a0013b0edba" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:30.087628 31992 raft_consensus.cc:3037] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.088258 29666 leader_election.cc:334] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.088660 31992 raft_consensus.cc:2454] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
W20240502 14:01:30.088729 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.088928 29666 leader_election.cc:302] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.089130 17212 raft_consensus.cc:2781] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.089244 17212 raft_consensus.cc:686] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:01:30.089397 17212 consensus_queue.cc:237] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:30.091567 29327 catalog_manager.cc:5484] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 2 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } }
W20240502 14:01:30.096549 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.107296 17019 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.107396 17019 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:30.107806 17019 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:30.109292 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:30.109411 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:30.109467 29404 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:30.109596 17019 raft_consensus.cc:2726] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:30.109616 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.110137 17210 raft_consensus.cc:482] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.110241 17210 raft_consensus.cc:504] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.110643 17210 leader_election.cc:288] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:30.110985 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631725c869674c45ac79c3f05cee8356" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:30.111122 29471 raft_consensus.cc:2454] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:30.111371 29799 leader_election.cc:302] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:01:30.111559 17210 raft_consensus.cc:2781] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.111616 17210 raft_consensus.cc:482] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.111667 17210 raft_consensus.cc:3037] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.111932 29797 leader_election.cc:334] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.112926 17210 raft_consensus.cc:504] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.113231 17210 leader_election.cc:288] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:30.114022 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631725c869674c45ac79c3f05cee8356" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:30.114143 29471 raft_consensus.cc:3037] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.114321 29797 leader_election.cc:334] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:30.114431 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.115247 29471 raft_consensus.cc:2454] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:01:30.115494 29799 leader_election.cc:302] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.115682 17210 raft_consensus.cc:2781] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.115782 17210 raft_consensus.cc:686] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:30.115917 17210 consensus_queue.cc:237] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.118006 29327 catalog_manager.cc:5484] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
I20240502 14:01:30.142210 17210 raft_consensus.cc:482] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.142324 17210 raft_consensus.cc:504] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.142982 17210 leader_election.cc:288] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:30.143185 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69badccfbc416f92d982cf394ea0cd" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:30.143306 29732 raft_consensus.cc:2454] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:30.143594 29798 leader_election.cc:302] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:30.143745 17210 raft_consensus.cc:2781] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.143790 17210 raft_consensus.cc:482] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.143831 17210 raft_consensus.cc:3037] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.143851 29797 leader_election.cc:334] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.144651 17210 raft_consensus.cc:504] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.144949 17210 leader_election.cc:288] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:30.145251 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69badccfbc416f92d982cf394ea0cd" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:30.145360 29732 raft_consensus.cc:3037] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.145880 29797 leader_election.cc:334] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.146338 29732 raft_consensus.cc:2454] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:01:30.146553 17226 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.146638 17226 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.146943 17226 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.146975 29798 leader_election.cc:302] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.147364 17210 raft_consensus.cc:2781] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.147567 17210 raft_consensus.cc:686] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:30.147751 17210 consensus_queue.cc:237] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.148550 17212 raft_consensus.cc:482] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.148643 17212 raft_consensus.cc:504] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
W20240502 14:01:30.149212 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:30.149672 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:30.149771 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.149771 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c6786a80e40a98130a5d9057e2877" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:30.149816 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.149912 31992 raft_consensus.cc:2454] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:01:30.149950 17226 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.150866 29327 catalog_manager.cc:5484] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:30.149457 17212 leader_election.cc:288] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
W20240502 14:01:30.151468 29666 leader_election.cc:334] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.151674 29666 leader_election.cc:302] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.151827 17212 raft_consensus.cc:2781] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.151882 17212 raft_consensus.cc:482] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.151935 17212 raft_consensus.cc:3037] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.152979 17212 raft_consensus.cc:504] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.153311 17212 leader_election.cc:288] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Requested vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:30.153837 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c6786a80e40a98130a5d9057e2877" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:30.153966 31990 raft_consensus.cc:3037] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.154767 31990 raft_consensus.cc:2454] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
I20240502 14:01:30.155040 29666 leader_election.cc:302] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:30.155215 17212 raft_consensus.cc:2781] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.155301 17212 raft_consensus.cc:686] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:01:30.155860 17212 consensus_queue.cc:237] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
W20240502 14:01:30.156006 29666 leader_election.cc:334] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.158380 29327 catalog_manager.cc:5484] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 2 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
W20240502 14:01:30.169072 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.169315 17226 raft_consensus.cc:482] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.169405 17226 raft_consensus.cc:504] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.169757 17226 leader_election.cc:288] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201)
I20240502 14:01:30.170209 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19374a33aaf41069e98fd035d2b9f00" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
W20240502 14:01:30.171062 29404 leader_election.cc:334] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.171121 29404 leader_election.cc:302] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: 8973f2527acb48f5ad36589652de73e1, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.171249 17226 raft_consensus.cc:2726] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.189518 17212 raft_consensus.cc:482] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.189635 17212 raft_consensus.cc:504] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.190080 17212 leader_election.cc:288] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:30.190398 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e9d8ef4f754df3817cc4a981549e02" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:30.190541 32254 raft_consensus.cc:2454] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:01:30.190776 29668 leader_election.cc:302] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:30.190936 17212 raft_consensus.cc:2781] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.190987 17212 raft_consensus.cc:482] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.191031 17212 raft_consensus.cc:3037] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.191169 29666 leader_election.cc:334] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.191742 17163 raft_consensus.cc:482] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.191850 17163 raft_consensus.cc:504] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.192165 17163 leader_election.cc:288] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:30.192165 17212 raft_consensus.cc:504] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.192512 17212 leader_election.cc:288] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Requested vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:30.192569 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f48a525d8c4a0783f0450aa6ec7cb0" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:30.193326 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e9d8ef4f754df3817cc4a981549e02" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1"
I20240502 14:01:30.193451 32254 raft_consensus.cc:3037] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.194010 29666 leader_election.cc:334] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.194413 32254 raft_consensus.cc:2454] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 2.
I20240502 14:01:30.194799 29668 leader_election.cc:302] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:30.195194 31925 leader_election.cc:334] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.195273 31925 leader_election.cc:302] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.195483 17163 raft_consensus.cc:2726] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.195595 17212 raft_consensus.cc:2781] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.195699 17212 raft_consensus.cc:686] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 LEADER]: Becoming Leader. State: Replica: efdbf7bdfe0b41bdadb1add053b09a7e, State: Running, Role: LEADER
I20240502 14:01:30.195838 17212 consensus_queue.cc:237] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.198426 29327 catalog_manager.cc:5484] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131). New cstate: current_term: 2 leader_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } }
I20240502 14:01:30.206475 16747 raft_consensus.cc:482] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.206601 16747 raft_consensus.cc:504] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.207043 16747 leader_election.cc:288] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
I20240502 14:01:30.207373 16747 raft_consensus.cc:482] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.207468 16747 raft_consensus.cc:504] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.207463 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fefaee1ab4c4a77aa7ed79d6c2e8894" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:30.207608 29732 raft_consensus.cc:2454] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:30.207796 16747 leader_election.cc:288] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.207923 29798 leader_election.cc:302] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters:
I20240502 14:01:30.208124 16747 raft_consensus.cc:2781] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.208184 16747 raft_consensus.cc:482] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.208153 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f316f08f5041a4907ad768c28ae30f" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:30.208230 16747 raft_consensus.cc:3037] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.208344 17210 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.208421 17210 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.208750 17210 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:30.208849 29797 leader_election.cc:334] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.209481 16747 raft_consensus.cc:504] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.209805 16747 leader_election.cc:288] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:30.209868 29797 leader_election.cc:334] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.209919 29797 leader_election.cc:302] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 269b366a09ff435ea8ec8dc01d0f9f6e, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.210033 16747 raft_consensus.cc:2726] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.210378 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fefaee1ab4c4a77aa7ed79d6c2e8894" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e"
I20240502 14:01:30.210508 29732 raft_consensus.cc:3037] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.210765 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:30.210942 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:30.211004 29797 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:30.211123 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.211133 16747 raft_consensus.cc:2726] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:30.211634 29797 leader_election.cc:334] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.211705 29732 raft_consensus.cc:2454] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
I20240502 14:01:30.211988 29798 leader_election.cc:302] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.212206 16747 raft_consensus.cc:2781] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.212296 16747 raft_consensus.cc:686] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:30.212445 16747 consensus_queue.cc:237] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.214979 29327 catalog_manager.cc:5484] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:30.215730 17163 raft_consensus.cc:482] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.215823 17163 raft_consensus.cc:504] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.216174 17163 leader_election.cc:288] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.216459 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f316f08f5041a4907ad768c28ae30f" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:30.216558 29864 raft_consensus.cc:2454] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:01:30.216956 31926 leader_election.cc:302] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:01:30.217180 17163 raft_consensus.cc:2781] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.217242 17163 raft_consensus.cc:482] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.217295 17163 raft_consensus.cc:3037] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.218050 17163 raft_consensus.cc:504] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.218320 17163 leader_election.cc:288] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.218420 17210 raft_consensus.cc:482] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.218505 17210 raft_consensus.cc:504] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.218797 17210 leader_election.cc:288] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:30.218781 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f316f08f5041a4907ad768c28ae30f" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:01:30.218964 29864 raft_consensus.cc:3037] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.219811 29797 leader_election.cc:334] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:30.219899 31925 leader_election.cc:334] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:30.220026 31925 leader_election.cc:334] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.220023 29864 raft_consensus.cc:2454] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:30.220327 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f48a525d8c4a0783f0450aa6ec7cb0" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:30.220440 31990 raft_consensus.cc:2454] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
I20240502 14:01:30.220604 31926 leader_election.cc:302] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.220685 29797 leader_election.cc:302] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.220824 17210 raft_consensus.cc:2781] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.220880 17210 raft_consensus.cc:482] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.220934 17210 raft_consensus.cc:3037] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.221004 17163 raft_consensus.cc:2781] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.221112 17163 raft_consensus.cc:686] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:01:30.221616 17163 consensus_queue.cc:237] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:30.221925 17212 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:30.221928 17210 raft_consensus.cc:504] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.222108 17212 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:30.222236 17210 leader_election.cc:288] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137)
I20240502 14:01:30.223135 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:30.223263 32254 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
I20240502 14:01:30.223513 17212 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:30.224053 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.224867 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f48a525d8c4a0783f0450aa6ec7cb0" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e"
I20240502 14:01:30.224983 31990 raft_consensus.cc:3037] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.225612 29327 catalog_manager.cc:5484] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } }
I20240502 14:01:30.225915 31990 raft_consensus.cc:2454] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 2.
W20240502 14:01:30.226241 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:30.226296 29666 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:30.226514 17212 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.227162 29797 leader_election.cc:302] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:01:30.227304 17210 raft_consensus.cc:2781] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 14:01:30.227310 29797 leader_election.cc:334] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.227402 17210 raft_consensus.cc:686] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 LEADER]: Becoming Leader. State: Replica: 4aebe6d7a3f34bf7b2bbbfbef1077463, State: Running, Role: LEADER
I20240502 14:01:30.227519 17210 consensus_queue.cc:237] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
W20240502 14:01:30.229298 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:30.229658 29327 catalog_manager.cc:5484] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132). New cstate: current_term: 2 leader_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
W20240502 14:01:30.230964 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.233034 17163 raft_consensus.cc:482] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.233152 17163 raft_consensus.cc:504] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.233448 17163 leader_election.cc:288] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:30.233805 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2ab852e31d4181a8597b9047a08554" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:30.233949 29471 raft_consensus.cc:2454] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
I20240502 14:01:30.234210 31927 leader_election.cc:302] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters:
W20240502 14:01:30.234315 31925 leader_election.cc:334] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.234370 17163 raft_consensus.cc:2781] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.234426 17163 raft_consensus.cc:482] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.234465 17163 raft_consensus.cc:3037] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.235244 17163 raft_consensus.cc:504] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.235522 17163 leader_election.cc:288] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:30.235841 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2ab852e31d4181a8597b9047a08554" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:30.235949 29471 raft_consensus.cc:3037] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.236557 31925 leader_election.cc:334] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.236867 29471 raft_consensus.cc:2454] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 2.
I20240502 14:01:30.237126 31927 leader_election.cc:302] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.237272 17163 raft_consensus.cc:2781] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.237361 17163 raft_consensus.cc:686] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 LEADER]: Becoming Leader. State: Replica: 269b366a09ff435ea8ec8dc01d0f9f6e, State: Running, Role: LEADER
I20240502 14:01:30.237484 17163 consensus_queue.cc:237] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:30.239974 29327 catalog_manager.cc:5484] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134). New cstate: current_term: 2 leader_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } }
W20240502 14:01:30.246510 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.250808 17210 raft_consensus.cc:482] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.250932 17210 raft_consensus.cc:504] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.251250 17210 leader_election.cc:288] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.251518 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e9f5068aa48b5a87841bdf720d9e0" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
W20240502 14:01:30.252099 29797 leader_election.cc:334] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.252158 29797 leader_election.cc:302] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: 1d00d3a47c52478894cf8c66959ef7bd, d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.252300 17210 raft_consensus.cc:2726] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:30.257144 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:30.280637 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.289227 17226 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.289342 17226 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:30.289700 17226 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:30.290678 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:30.290796 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.290859 29404 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.291007 17226 raft_consensus.cc:2726] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:30.291412 16508 raft_consensus.cc:482] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.291508 16508 raft_consensus.cc:504] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.291867 16508 leader_election.cc:288] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:30.292058 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.292062 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c91754a477542e5abea67a09971d111" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:30.292181 29864 raft_consensus.cc:2454] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:30.292426 32189 leader_election.cc:302] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:30.292569 16508 raft_consensus.cc:2781] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.292620 16508 raft_consensus.cc:482] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.292675 16508 raft_consensus.cc:3037] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.293157 32188 leader_election.cc:334] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.293702 16508 raft_consensus.cc:504] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.294010 16508 leader_election.cc:288] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.294255 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c91754a477542e5abea67a09971d111" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:01:30.294365 29864 raft_consensus.cc:3037] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.294832 32188 leader_election.cc:334] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.295203 29864 raft_consensus.cc:2454] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:30.295483 32189 leader_election.cc:302] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.295681 16508 raft_consensus.cc:2781] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.295758 16508 raft_consensus.cc:686] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:30.295893 16508 consensus_queue.cc:237] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.298076 29327 catalog_manager.cc:5484] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
I20240502 14:01:30.304185 17163 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.304276 17163 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:30.304594 17163 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:30.305490 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:30.305589 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:30.305639 31925 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.305784 17163 raft_consensus.cc:2726] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:30.316535 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.337729 16508 raft_consensus.cc:482] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.337833 16508 raft_consensus.cc:504] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.338202 16508 leader_election.cc:288] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.338505 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19374a33aaf41069e98fd035d2b9f00" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:30.338645 29471 raft_consensus.cc:2454] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:30.338943 32190 leader_election.cc:302] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters:
I20240502 14:01:30.339082 16508 raft_consensus.cc:2781] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 14:01:30.339191 32188 leader_election.cc:334] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.339147 16508 raft_consensus.cc:482] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.339346 16508 raft_consensus.cc:3037] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.340396 16508 raft_consensus.cc:504] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.340692 16508 leader_election.cc:288] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Requested vote from peers 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
I20240502 14:01:30.340962 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19374a33aaf41069e98fd035d2b9f00" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd"
I20240502 14:01:30.341064 29471 raft_consensus.cc:3037] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.341635 32188 leader_election.cc:334] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.341961 29471 raft_consensus.cc:2454] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 2.
I20240502 14:01:30.342211 32190 leader_election.cc:302] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.342347 16508 raft_consensus.cc:2781] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.342430 16508 raft_consensus.cc:686] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [term 2 LEADER]: Becoming Leader. State: Replica: 8973f2527acb48f5ad36589652de73e1, State: Running, Role: LEADER
I20240502 14:01:30.342566 16508 consensus_queue.cc:237] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.344867 29327 catalog_manager.cc:5484] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 8973f2527acb48f5ad36589652de73e1 (127.22.55.136). New cstate: current_term: 2 leader_uuid: "8973f2527acb48f5ad36589652de73e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } }
W20240502 14:01:30.349712 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:30.365810 17226 raft_consensus.cc:482] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.365919 17226 raft_consensus.cc:504] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.366349 17226 leader_election.cc:288] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:30.366734 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e9f5068aa48b5a87841bdf720d9e0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:30.366900 29864 raft_consensus.cc:2454] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:01:30.367148 29405 leader_election.cc:302] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters:
I20240502 14:01:30.367290 17226 raft_consensus.cc:2781] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:01:30.367348 17226 raft_consensus.cc:482] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.367389 17226 raft_consensus.cc:3037] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:01:30.367774 29404 leader_election.cc:334] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.368621 17226 raft_consensus.cc:504] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:30.368973 17226 leader_election.cc:288] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: Requested vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
W20240502 14:01:30.369622 29404 leader_election.cc:334] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.369788 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28e9f5068aa48b5a87841bdf720d9e0" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463"
I20240502 14:01:30.369926 29864 raft_consensus.cc:3037] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.370688 29864 raft_consensus.cc:2454] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 2.
I20240502 14:01:30.370965 29405 leader_election.cc:302] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423
I20240502 14:01:30.371192 17226 raft_consensus.cc:2781] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.371271 17226 raft_consensus.cc:686] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 LEADER]: Becoming Leader. State: Replica: 1d00d3a47c52478894cf8c66959ef7bd, State: Running, Role: LEADER
I20240502 14:01:30.371408 17226 consensus_queue.cc:237] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
W20240502 14:01:30.372938 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.373742 29327 catalog_manager.cc:5484] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd reported cstate change: term changed from 1 to 2, leader changed from d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130) to 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129). New cstate: current_term: 2 leader_uuid: "1d00d3a47c52478894cf8c66959ef7bd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } }
I20240502 14:01:30.380496 16508 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:30.380601 16508 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:30.380935 16508 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:30.382004 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:30.382107 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.382158 32188 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:30.382287 16508 raft_consensus.cc:2726] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:30.396171 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:30.401286 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:30.449350 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.453202 32254 raft_consensus.cc:1261] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.453600 17163 consensus_queue.cc:1035] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:30.456630 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.467676 29471 raft_consensus.cc:1261] T 835183257412496b90defa7ecaa0ff13 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:30.468041 16508 consensus_queue.cc:1035] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:30.503568 31990 raft_consensus.cc:1261] T f716493d95f84740beb85c3bf4021ad0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:30.503996 17019 consensus_queue.cc:1035] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:30.507213 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:30.507373 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:30.515664 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.528533 29469 raft_consensus.cc:1261] T 631725c869674c45ac79c3f05cee8356 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.528915 17210 consensus_queue.cc:1035] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:30.538547 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:30.539518 31990 raft_consensus.cc:1261] T df1c6786a80e40a98130a5d9057e2877 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.539852 16482 consensus_queue.cc:1035] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:30.544003 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.576897 29733 raft_consensus.cc:1261] T 4e69badccfbc416f92d982cf394ea0cd P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.577313 17210 consensus_queue.cc:1035] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:30.580454 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.592173 31990 raft_consensus.cc:1261] T 891851115e2e4d9db5bf3a0013b0edba P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:30.592581 16482 consensus_queue.cc:1035] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:30.595607 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:30.618326 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:30.622695 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:30.646425 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:30.646798 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:30.648933 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:30.673014 32254 raft_consensus.cc:1261] T 27e9d8ef4f754df3817cc4a981549e02 P 8973f2527acb48f5ad36589652de73e1 [term 2 FOLLOWER]: Refusing update from remote peer efdbf7bdfe0b41bdadb1add053b09a7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.673404 17212 consensus_queue.cc:1035] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:30.674054 29471 raft_consensus.cc:1261] T 4a2ab852e31d4181a8597b9047a08554 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.674481 17163 consensus_queue.cc:1035] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:30.678802 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.694906 31990 raft_consensus.cc:1261] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.695341 17210 consensus_queue.cc:1035] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:30.699092 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.711077 29733 raft_consensus.cc:1261] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P efdbf7bdfe0b41bdadb1add053b09a7e [term 2 FOLLOWER]: Refusing update from remote peer 4aebe6d7a3f34bf7b2bbbfbef1077463: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:30.711448 17210 consensus_queue.cc:1035] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:30.721331 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:30.759218 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.760677 29471 raft_consensus.cc:1261] T b19374a33aaf41069e98fd035d2b9f00 P 1d00d3a47c52478894cf8c66959ef7bd [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:30.761021 17218 consensus_queue.cc:1035] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:01:30.763782 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.786875 29864 raft_consensus.cc:1261] T f28e9f5068aa48b5a87841bdf720d9e0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer 1d00d3a47c52478894cf8c66959ef7bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:01:30.787282 17226 consensus_queue.cc:1035] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:30.790155 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.809172 29864 raft_consensus.cc:1261] T f5f316f08f5041a4907ad768c28ae30f P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer 269b366a09ff435ea8ec8dc01d0f9f6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:30.809561 17163 consensus_queue.cc:1035] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:30.813853 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:30.846467 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:30.885496 29863 raft_consensus.cc:1261] T 1c91754a477542e5abea67a09971d111 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Refusing update from remote peer 8973f2527acb48f5ad36589652de73e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:01:30.885919 16508 consensus_queue.cc:1035] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:01:30.943483 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.003970 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:31.004108 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.008888 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.042925 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.059864 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:31.091919 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.120926 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.133605 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.178964 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.193862 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.220311 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.274057 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.290956 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.303851 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.305199 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.309201 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:31.314532 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.345185 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.369457 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.369941 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.374233 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.410102 17224 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.410218 17224 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:31.410617 17224 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:31.411646 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.411741 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:31.411785 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:31.411912 17224 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:31.411919 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:31.415844 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.438671 17210 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.438802 17210 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:31.439234 17210 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:31.440277 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.440358 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:31.440392 29797 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:31.440496 17210 raft_consensus.cc:2726] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:31.450114 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.450863 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.453606 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.459650 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.463217 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.493727 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.499191 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:31.504572 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.521054 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.540607 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:31.557844 17224 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:31.557986 17224 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:31.558377 17224 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:31.559240 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.559597 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.559710 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:31.559770 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:31.559973 17224 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.560298 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.560847 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.565353 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.567823 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.579464 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.587466 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:31.588886 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.595280 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.607461 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.608948 16508 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.609053 16508 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:31.609442 16508 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:31.610347 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.610466 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:31.610509 32188 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:31.610623 16508 raft_consensus.cc:2726] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:31.616765 17224 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.616866 17224 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:31.617244 17224 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:31.617621 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:31.617758 29471 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:31.618295 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.618391 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:31.618443 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:31.618566 17224 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.622264 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.624130 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.626950 17263 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.627030 17263 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:31.627372 17263 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:31.628449 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.628569 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:31.628623 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:31.628768 17263 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.631731 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.649416 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.660127 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.666595 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.667337 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.683804 17163 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.683910 17163 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:31.684265 17163 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:31.685254 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:31.685370 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:31.685412 31925 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:31.685539 17163 raft_consensus.cc:2726] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.690661 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.691219 17265 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:31.691303 17265 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:31.691637 17265 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:31.692616 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:31.692731 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:31.692795 29797 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:31.693114 17265 raft_consensus.cc:2726] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.693589 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.696589 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.707414 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.722061 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.728811 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.732038 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.742127 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.760203 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.768566 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.783030 17218 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.783128 17218 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:31.783499 17218 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:31.784299 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.784377 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:31.784425 32188 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:31.784538 17218 raft_consensus.cc:2726] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.816267 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.816294 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.852023 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:31.859467 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.901878 17218 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:31.902025 17218 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:31.902570 17218 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:31.902781 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:31.902962 29864 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:31.903134 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:31.903856 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:31.903863 16482 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:31.903926 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:31.903955 16482 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:31.904055 17218 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:31.904320 16482 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:31.904816 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.905862 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:31.906004 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:31.906065 29666 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:31.906263 16482 raft_consensus.cc:2726] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:31.908586 17265 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:31.908725 17265 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:31.909127 17265 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:31.910718 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:31.910830 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:31.910957 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:31.911212 17265 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:31.930233 17265 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:31.930363 17265 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:31.930747 17265 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:31.931887 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:31.932548 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:31.932631 29797 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:31.932771 17265 raft_consensus.cc:2726] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.941005 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:31.941978 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:31.949007 17163 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:31.949137 17163 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:31.949636 17163 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:31.952652 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:31.952775 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:31.952826 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:31.952951 17163 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:31.953096 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:31.967785 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:31.973634 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:32.009999 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:32.012894 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:01:32.016559 17265 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:32.016671 17265 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:32.017047 17265 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:32.018003 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:32.018121 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:32.018164 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:32.018293 17265 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:32.028535 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.043826 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:32.055578 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:32.090982 17263 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:32.091110 17263 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:32.091450 17263 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:32.092360 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:32.092490 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:32.092546 29404 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:32.092700 17263 raft_consensus.cc:2726] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:32.097947 17226 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:32.098031 17226 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:32.098364 17226 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:32.099619 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:32.099710 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:32.099764 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:32.099893 17226 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:32.102135 17265 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:32.102236 17265 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:32.102686 17265 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:32.103891 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:32.103994 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:32.104045 29797 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:32.104176 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:32.104184 17265 raft_consensus.cc:2726] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:32.109691 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:32.122448 17226 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:32.122545 17226 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:32.122995 17226 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:32.124449 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:32.124552 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:32.124598 29404 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:32.124718 17226 raft_consensus.cc:2726] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:32.124738 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:32.139551 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:32.151670 17163 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:32.151764 17163 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:32.152097 17163 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:32.153092 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:32.153218 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:32.153273 31925 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:32.153396 17163 raft_consensus.cc:2726] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:32.184810 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:32.206079 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:32.212558 17163 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:32.212699 17163 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:32.213107 17163 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:32.213291 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:32.214062 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:32.214174 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:32.214222 31925 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:32.214381 17163 raft_consensus.cc:2726] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:32.246176 17265 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:32.246279 17265 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:32.246564 17265 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:32.248278 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:32.248402 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:32.248458 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:32.248581 17210 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:32.300700 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.319145 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:32.401630 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:32.418680 17249 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:32.418784 17249 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:32.419266 17249 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:32.419500 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:32.419677 32254 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
W20240502 14:01:32.420603 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:32.420714 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:32.420759 29666 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:32.420883 17249 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:32.494980 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:32.511494 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.520274 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:32.530164 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.591430 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.592187 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:32.612421 17218 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:32.612547 17218 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:32.612939 17218 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:32.613857 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:32.613940 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:32.613991 32188 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:32.614132 17218 raft_consensus.cc:2726] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:32.615917 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.646603 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:32.708667 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:32.710276 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:32.719817 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:32.756309 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.783793 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:32.822291 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.824060 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.844468 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.901093 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.902900 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:32.905810 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.951375 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:32.972455 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:32.979529 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:33.011888 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.017026 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.097618 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.116190 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.117266 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.119648 17163 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:33.119760 17163 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:33.120229 17163 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:33.120579 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:33.120728 29471 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:33.121176 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:33.121285 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:33.121336 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:33.121462 17163 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:33.124763 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.143412 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.150568 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.151608 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.182569 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 291 similar messages]
W20240502 14:01:33.187067 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:33.187181 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.195704 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.221704 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.240635 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:33.261410 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:33.279166 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.279660 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 242 similar messages]
W20240502 14:01:33.281319 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.315936 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.333578 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.353821 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 205 similar messages]
W20240502 14:01:33.365914 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.373261 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.374889 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 251 similar messages]
W20240502 14:01:33.380750 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.386330 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.394285 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:33.398485 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.465698 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:33.494643 17269 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:33.494748 17269 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:33.495211 17269 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:33.496501 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:33.496590 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:33.496640 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:33.496767 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:33.496781 17269 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:33.510195 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.530787 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 14:01:33.550194 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.594574 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:33.624056 17269 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:33.624182 17269 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:33.624521 17269 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:33.625530 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:33.625648 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:33.625705 31925 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:33.625844 17269 raft_consensus.cc:2726] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:33.671949 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.694207 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:33.699295 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.734524 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.752188 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.771914 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.780170 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.796036 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.797617 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.801887 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.811143 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:33.825445 17210 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:33.825536 17210 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:33.825858 17210 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:33.826761 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:33.826891 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:33.826949 29797 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:33.827093 17210 raft_consensus.cc:2726] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:33.831127 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.856855 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.866874 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.868357 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.881943 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:33.898345 17273 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:33.898455 17273 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:33.898864 17273 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:33.899844 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:33.899945 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:33.899997 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:33.900147 17273 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:33.904449 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.909003 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.946106 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:33.956517 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.968681 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:33.970778 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:33.976995 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:33.987018 17218 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:33.987107 17218 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:33.987466 17218 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:33.987792 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:33.987911 29732 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:33.988389 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:33.988520 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:33.988575 32188 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:33.988715 17218 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.024574 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.054610 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.057571 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.062745 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.079321 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.079795 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.088032 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:34.091202 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.091552 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.099052 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.107273 17249 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.107375 17249 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:34.107687 17249 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.108565 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:34.108650 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.108700 29666 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:34.108875 17249 raft_consensus.cc:2726] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:34.108872 17210 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:34.108960 17210 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:34.109335 17210 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.110241 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.110324 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.110373 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.110509 17210 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.123641 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.133080 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.148113 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.151044 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.152498 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.153473 17279 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.153561 17279 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:34.153934 17279 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.154990 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.155097 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.155159 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.155359 17279 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.164187 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:34.186372 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.201983 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.207356 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.214488 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.221833 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.241758 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.244794 17210 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.244890 17210 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:34.245306 17210 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:34.246188 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.246255 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:34.246384 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:34.246449 29797 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.246578 17210 raft_consensus.cc:2726] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.264542 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.285291 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:34.313637 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.337606 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.347011 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:34.373607 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.376379 17218 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:34.376471 17218 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:34.376888 17218 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.378046 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.378155 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.378207 32188 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.378330 17218 raft_consensus.cc:2726] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.379726 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.380501 17269 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.380590 17269 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:34.380992 17269 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:34.382207 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.382302 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.382359 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.382472 17269 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.382557 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:34.389788 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.422071 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.438961 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:34.447494 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.452052 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.465018 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.471117 17210 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:34.471225 17210 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:34.471588 17210 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:34.472460 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.472858 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:34.472914 29797 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:34.473039 17210 raft_consensus.cc:2726] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:34.481750 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.484186 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:34.485255 17210 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:34.485358 17210 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:34.485767 17210 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.486961 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:34.487071 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:34.487131 29797 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:34.487233 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.487260 17210 raft_consensus.cc:2726] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:34.491125 17269 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.491214 17269 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:34.491537 17269 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.492395 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.492511 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.493952 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.494094 17269 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.507059 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.510273 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:34.533000 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.566210 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.573459 17273 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.573576 17273 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:34.573989 17273 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:34.575383 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:34.575488 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:34.575541 29404 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:34.575666 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:34.575773 17273 raft_consensus.cc:2726] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.580045 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:34.608029 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:34.633219 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:34.654248 17273 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.654356 17273 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:34.654685 17273 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:34.655647 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:34.655773 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:34.655835 29404 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.655962 17273 raft_consensus.cc:2726] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.683378 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:01:34.688707 17218 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.688835 17218 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:34.689183 17218 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.690369 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.690451 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.690485 32188 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.690589 17274 raft_consensus.cc:2726] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.698959 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:34.714653 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:34.719509 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:34.720489 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:34.787925 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:34.865777 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:01:34.879382 17210 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.879491 17210 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:34.879846 17210 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.880815 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:34.881242 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.881304 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:34.881454 17210 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:34.881706 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:34.891777 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:34.920388 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:34.921223 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:01:34.936134 17218 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:34.936249 17218 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:34.936640 17218 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:34.937661 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.937814 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:34.937870 32188 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.938002 17274 raft_consensus.cc:2726] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:34.962653 17273 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:34.962780 17273 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:34.963227 17273 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:34.963846 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:34.963889 17269 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:34.963994 31990 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:01:34.963980 17269 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:34.964279 17269 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:34.964613 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:34.964779 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:34.964840 29404 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:34.965219 17273 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:34.965684 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:34.965783 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:34.965829 31925 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:34.966204 17288 raft_consensus.cc:2726] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:34.988673 17284 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:34.988790 17284 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:34.989156 17284 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:34.990221 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:34.990366 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:34.990428 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:34.990546 17284 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:35.020568 17288 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:35.020678 17288 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:35.020993 17288 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:35.021921 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:35.022018 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:35.022065 31925 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:35.022212 17288 raft_consensus.cc:2726] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:35.080708 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.086722 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:35.088937 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.094529 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:35.146039 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.261864 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.263903 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.301801 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.336218 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:35.357509 17210 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:35.357601 17210 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:35.357975 17210 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:35.359231 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:35.359345 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:35.359400 29797 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:35.359556 17210 raft_consensus.cc:2726] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:35.368046 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:35.368045 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:35.372822 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:35.428274 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:35.433274 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.456404 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.480235 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:35.491986 17273 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:35.492103 17273 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:35.492449 17273 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:35.493330 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:35.493422 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:35.493474 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:35.493606 17273 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:35.511444 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.528139 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.531057 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:35.539671 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.547557 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.589233 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:35.590001 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.594622 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.623786 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:35.634135 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.636175 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:35.638671 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.694530 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.721158 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.730448 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.740350 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.742885 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.748442 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:35.754735 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:35.760242 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:35.774435 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.777245 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.777678 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:35.786094 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:35.809633 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.846956 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:35.869908 17249 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:35.870028 17249 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:35.870417 17249 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:35.870730 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:35.870882 32255 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
W20240502 14:01:35.871686 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:35.871793 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:35.871843 29666 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:35.871986 17249 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:35.873049 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.904947 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:35.967221 17294 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:35.967379 17294 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:35.967880 17294 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:35.968924 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:35.969041 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:35.969098 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:35.969244 17294 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:35.971597 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.994496 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:36.020079 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.031903 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.100385 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:36.143328 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.165098 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:36.166179 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.189337 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.202608 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:36.221591 17269 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:36.221751 17269 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:36.222102 17269 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:36.223162 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:36.223423 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:36.223476 31925 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:36.223598 17269 raft_consensus.cc:2726] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:36.229914 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.230690 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.253710 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.263646 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.293246 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.302906 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.305811 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.321697 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.326145 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.346691 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:36.351991 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.366779 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.377593 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.412833 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.420042 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.461985 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.465787 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.469008 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.471344 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.476962 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:36.511334 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:01:36.539844 17273 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:36.539954 17273 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:36.540350 17273 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:36.540840 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:36.540977 31992 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:36.541527 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:36.541934 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:36.541993 29404 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:36.542162 17273 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:36.543478 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.548511 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.548897 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.551062 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.552384 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.558363 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:36.589839 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.597783 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.622725 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:36.666746 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.680075 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.698434 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:36.700242 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.704335 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.706445 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.750357 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.782744 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.812867 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:36.828742 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.831254 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:36.834117 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.874333 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.889526 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.911880 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.926311 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.945158 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:36.958776 17288 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:36.958895 17288 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:36.959262 17288 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:36.960191 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:36.960306 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:36.960347 31925 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:36.960464 17288 raft_consensus.cc:2726] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:36.971087 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:36.989472 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:36.991084 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:36.996974 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:37.011646 17298 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:37.011734 17298 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:37.012053 17298 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:37.013095 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:37.013213 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:37.013258 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:37.013399 17298 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:37.020632 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.033881 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.035974 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.036608 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:37.053468 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.054934 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.056947 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.075871 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.078519 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.079164 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:37.142446 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:37.182595 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.184141 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.262642 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:37.306787 17300 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:37.306936 17300 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:37.307353 17300 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:37.308274 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.308660 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:37.308806 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:37.308853 32188 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:37.308959 17300 raft_consensus.cc:2726] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:37.315440 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:37.335965 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:37.342470 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:37.364564 17298 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:37.364671 17298 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:37.365000 17298 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:37.365933 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:37.366019 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:37.366050 29797 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:37.366181 17298 raft_consensus.cc:2726] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:37.370800 17298 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:37.370901 17298 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:37.371261 17298 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:37.372509 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:37.372613 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:37.372668 29797 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:37.372788 17298 raft_consensus.cc:2726] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:37.372800 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:37.373953 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:01:37.392729 17302 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:37.392840 17302 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:37.393126 17302 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:37.394001 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:37.394109 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:37.394161 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:37.394307 17302 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:37.397050 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:37.482722 17302 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:37.482873 17302 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:37.483213 17302 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:37.484109 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:37.484251 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:37.484308 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:37.484444 17302 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:37.532732 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.539161 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:37.546406 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.564040 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.581305 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:37.610805 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:01:37.650777 17218 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:37.650907 17218 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:37.651264 17218 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:37.652294 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:37.652410 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:37.652454 32188 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:37.652571 17218 raft_consensus.cc:2726] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:37.653924 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:37.721642 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:37.741853 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:37.756700 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.770918 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.781894 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:37.840597 17303 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:37.840726 17303 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:37.841207 17303 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:37.841871 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.842676 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:37.842785 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:37.842865 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:37.842985 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:37.842991 17303 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:37.855382 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:37.953891 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.961824 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:37.987545 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.010619 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.019956 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.048918 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.065829 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:38.080619 17308 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:38.080721 17308 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
W20240502 14:01:38.081050 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.081113 17308 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:38.081952 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.082119 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:38.082181 32188 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:38.082347 17308 raft_consensus.cc:2726] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.083096 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.100672 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.104772 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:38.127352 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:38.131240 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.148234 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.183823 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.188913 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:38.196985 17288 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:38.197088 17288 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:38.197455 17288 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:38.198329 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.198432 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:38.198477 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:38.198602 17288 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.201711 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:38.207844 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 317 similar messages]
W20240502 14:01:38.208380 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:38.218042 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:38.230053 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.265089 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.306190 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 241 similar messages]
W20240502 14:01:38.306728 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.311007 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.311841 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.321573 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.328987 17288 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:38.329146 17288 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:38.329638 17288 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:38.330101 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:38.330246 29469 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:38.331161 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.331351 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:38.331444 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:38.331734 17288 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.377465 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 265 similar messages]
W20240502 14:01:38.388079 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 215 similar messages]
W20240502 14:01:38.420234 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.427579 17311 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:38.427672 17311 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:38.428028 17311 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:38.429200 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.429309 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:38.429363 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:38.429474 17311 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.443220 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.443222 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.444517 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:38.464653 17307 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:38.464778 17307 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:38.465159 17307 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:38.466071 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:38.466159 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:38.466192 29666 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:38.466297 17307 raft_consensus.cc:2726] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.468248 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.478049 17288 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:38.478148 17288 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:38.478561 17288 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:38.479624 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.479707 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:38.479754 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:38.479879 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:38.479895 17288 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.518210 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.539919 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:38.598014 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.607344 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.617063 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240502 14:01:38.634030 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.656463 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.660655 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.665230 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.668841 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.672341 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.675415 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.695834 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:38.703343 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.757575 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:38.760921 17311 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:38.761030 17311 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:38.761395 17311 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:38.762595 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:38.762708 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:38.762758 29797 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:38.762876 17311 raft_consensus.cc:2726] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:38.762980 17314 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:38.763083 17314 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:38.763448 17314 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:38.764714 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:38.764811 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:38.764853 29404 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:38.764969 17314 raft_consensus.cc:2726] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.765133 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.772450 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.786446 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.792292 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.818126 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:38.821010 17311 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:38.821120 17311 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:38.821480 17311 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:38.822382 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.822516 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:38.822569 29797 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:38.822678 17311 raft_consensus.cc:2726] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:38.873833 17308 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:38.873945 17308 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:38.874364 17308 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:38.874666 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:38.874799 29732 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:38.875556 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:38.875667 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:38.875713 32188 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:38.875841 17308 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:38.880455 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.880537 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:38.887063 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.888267 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.905510 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.908071 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.933115 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:38.977185 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.981555 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:38.995033 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:39.049358 17218 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:39.049470 17218 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:39.049999 17218 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:39.050251 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:39.050402 29469 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:39.050993 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:39.051175 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:39.051240 32188 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:39.051410 17218 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:39.052682 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.054534 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.073835 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.088306 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.098592 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.103304 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.105082 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.107218 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.109930 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:01:39.123368 17311 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:39.123476 17311 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:39.123811 17311 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:39.124647 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:39.124751 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:39.124799 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:39.124941 17311 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:39.139276 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.179162 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.196096 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.199324 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.213601 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.239246 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.249013 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:39.261723 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:39.267182 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:39.269249 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.272607 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.278203 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.312536 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.340867 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.354830 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.370291 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:39.373306 17318 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:39.373426 17318 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:39.373802 17318 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:39.374632 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:39.374733 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:39.374780 31925 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:39.374924 17318 raft_consensus.cc:2726] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:39.395603 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:39.397531 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:39.398080 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.410238 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.460902 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.466120 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.485416 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.510656 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.519788 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:39.521649 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.549131 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.564802 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.565968 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.586005 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.609269 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.616880 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:39.623008 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.639675 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:39.679311 17310 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:39.679445 17310 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:39.679838 17310 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:39.680766 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:39.680876 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:39.680927 29404 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:39.681051 17310 raft_consensus.cc:2726] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:39.688414 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.692137 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:39.751505 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.768553 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:39.824311 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:39.837859 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:39.846652 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:39.871577 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:39.882383 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.911716 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:39.921214 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:39.927276 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:39.988337 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.052342 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.062363 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.062840 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:40.088415 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.149416 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.149405 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.236706 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.319988 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.342989 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.351270 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:40.360054 17318 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:40.360165 17318 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:40.360584 17318 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:40.361945 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:40.362059 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:40.362116 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:40.362237 17318 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:40.362243 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:40.391690 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:40.399184 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:40.399372 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:40.433308 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.446058 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:40.449920 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:40.475634 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:01:40.510668 17218 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:40.510782 17218 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:40.511255 17218 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:40.511583 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:40.511713 29733 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:40.512681 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:40.512804 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:40.512861 32188 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:40.512993 17218 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:40.562498 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.564680 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.567828 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.596376 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.612654 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:40.650411 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:40.658215 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.664773 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.689088 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.719707 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.724366 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.733827 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.751215 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.796223 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:40.807340 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.832727 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.835798 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:40.846755 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.856948 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.879179 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:40.934067 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:40.944749 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:40.967378 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.972975 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:41.024925 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:41.049381 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:41.066831 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.069169 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.091838 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.107273 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:41.153364 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.174211 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:41.207391 17313 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:41.207504 17313 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:41.208009 17313 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:41.208351 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:41.208492 29471 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:41.209196 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:41.209311 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:41.209367 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:41.209437 17311 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:41.209482 17313 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:41.209512 17311 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:41.209827 17311 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:41.210670 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:41.210788 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:41.210835 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:41.210973 17311 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:41.213640 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.233260 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.234689 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.250715 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.255906 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.256127 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:41.275307 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.313666 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.329563 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.369069 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.393558 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.397460 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.398159 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.403882 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.404522 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.445605 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.447652 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.454183 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:41.471122 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.475679 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.482576 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:41.491776 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.510561 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.594516 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.625566 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:41.626679 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.645699 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.666128 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.668071 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:41.676225 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.689273 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.695962 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.697618 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.700825 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.703833 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.708613 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.722198 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:41.781560 17326 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:41.781682 17326 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:41.782012 17326 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:41.782284 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.783133 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:41.783265 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:41.783324 29797 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:41.783492 17326 raft_consensus.cc:2726] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:41.829994 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.846834 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.866189 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.878594 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.880802 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.883328 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.884001 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.889478 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:41.924340 17318 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:41.924486 17318 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:41.924885 17318 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:41.925896 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:41.926012 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:41.926054 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:41.926170 17318 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:41.929983 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:41.932523 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.938489 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:41.952096 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:41.966156 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:41.972096 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:41.984216 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:41.984514 17318 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:41.984625 17318 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:41.985020 17318 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:41.986167 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:41.986295 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:41.986340 31925 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:41.986506 17318 raft_consensus.cc:2726] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:41.994607 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:42.036077 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.048424 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:42.067416 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:42.089859 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:42.158097 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:42.173565 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.178889 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.180100 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.190816 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:42.206243 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.226145 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.232620 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.236627 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.250954 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.315282 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.321035 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:42.326224 17331 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:42.326360 17331 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:42.326706 17331 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:42.327781 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:42.327879 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:42.327921 29666 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:42.328042 17331 raft_consensus.cc:2726] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:42.356833 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:42.380241 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.392587 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.450867 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:42.490028 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.502259 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:42.508319 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.526950 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:42.544502 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:42.594983 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.631947 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.641785 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:42.668263 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:42.674749 17318 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:42.674839 17318 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:42.675207 17318 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:42.676100 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:42.676220 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:42.676265 31925 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:42.676396 17318 raft_consensus.cc:2726] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:42.684427 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.694806 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:01:42.748991 17334 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:42.749094 17334 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:42.749431 17334 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:42.749931 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:42.750072 31990 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:42.750716 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:42.750921 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:42.750983 29404 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:42.751117 17334 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:42.791076 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.819927 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:42.850194 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:42.877534 17336 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:42.877664 17336 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:42.878106 17336 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:42.878376 29863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:42.878513 29863 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:42.878659 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:42.879761 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:42.879838 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:42.880004 17336 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:42.894870 17334 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:42.894973 17334 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:42.895326 17334 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:42.896378 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:42.896549 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:42.896602 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:42.896741 17334 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:42.926667 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:42.992892 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:42.993291 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:01:43.001001 17338 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:43.001101 17338 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:43.001470 17338 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:43.002482 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:43.002589 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:43.002640 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:43.002789 17338 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:43.038228 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:43.049533 17338 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:43.049640 17338 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:43.050870 17338 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:43.051110 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:43.051250 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:43.051298 29797 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:43.051482 17338 raft_consensus.cc:2726] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:43.080206 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.131232 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:43.154359 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.162786 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.167905 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.188009 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.193819 17338 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:43.193936 17338 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:43.194300 17338 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:43.195394 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:43.195510 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:43.195564 29797 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:43.195694 17338 raft_consensus.cc:2726] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:43.210347 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.222198 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.228662 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 305 similar messages]
W20240502 14:01:43.260016 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.277663 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.282876 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.297683 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.300921 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.307252 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 240 similar messages]
W20240502 14:01:43.307876 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:43.317951 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.333918 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:43.348253 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.351682 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:01:43.355865 17339 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:43.355994 17339 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:43.356436 17339 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:43.356813 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.357452 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:43.357586 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:43.357630 29404 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:43.357745 17339 raft_consensus.cc:2726] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:43.379838 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 259 similar messages]
W20240502 14:01:43.380628 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.380784 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:01:43.381798 17336 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:43.381891 17336 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:43.382261 17336 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:43.383420 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:43.383528 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:43.383569 32188 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:43.383677 17336 raft_consensus.cc:2726] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:43.398375 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 215 similar messages]
W20240502 14:01:43.417965 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.425714 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:43.444902 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.450546 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.508754 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.511410 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.525538 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.551121 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.580111 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.582237 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.591307 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.598762 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.610586 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.626950 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.630573 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.633502 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240502 14:01:43.686002 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.719877 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:43.740355 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.746619 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:43.771521 17218 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:43.771629 17218 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:43.771999 17218 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:43.772311 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:43.772468 29471 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:43.772599 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:43.773016 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:43.773095 32188 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:43.773226 17218 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:43.776554 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:43.803753 17318 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:43.803877 17318 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:43.804339 17318 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:43.805617 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:43.805697 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:43.805743 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:43.805871 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:43.805886 17318 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:43.818344 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.822659 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.840369 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.844599 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.858058 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.862829 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.901191 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:43.915273 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.917676 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.944929 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.959221 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:43.976912 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:43.991329 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.000097 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.034860 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.045974 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.049835 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.067236 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.083526 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:44.086258 17345 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:44.086366 17345 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:44.086751 17345 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:44.087615 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:44.087728 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:44.087770 32188 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:44.087886 17345 raft_consensus.cc:2726] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:44.099912 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.110607 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.132438 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.151118 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:44.157703 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.161940 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:44.166944 17311 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:44.167028 17311 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:44.167429 17311 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:44.168673 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:44.168780 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:44.168835 29797 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:44.168949 17311 raft_consensus.cc:2726] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:44.168968 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:44.184137 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.186000 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:44.200439 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:44.208457 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:44.211138 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.250838 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.253338 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.305656 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.374925 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.379287 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.389430 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:44.408677 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.411096 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.431989 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.441601 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:44.449586 17336 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:44.449687 17336 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:44.450143 17336 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:44.450529 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:44.450672 29733 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:44.451320 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:44.451438 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:44.451515 32188 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:44.451655 17336 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:44.453617 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.503118 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:44.517711 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.519109 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.520862 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:44.539561 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:44.550604 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.560879 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.571808 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.584581 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.588038 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.598485 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:44.609180 17345 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:44.609283 17345 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:44.609669 17345 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:44.611414 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:44.612224 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:44.612298 32188 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:44.612427 17345 raft_consensus.cc:2726] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:44.668998 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:01:44.690142 17318 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:44.690282 17318 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:44.690894 17318 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:44.691061 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:44.691197 29469 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:44.692080 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:44.692202 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:44.692260 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:44.692454 17318 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:44.745404 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.747542 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.751289 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:44.763433 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:44.777153 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.792678 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.799491 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.834197 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.849749 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.876369 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:44.882812 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:44.884038 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:44.890441 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.899322 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:44.972805 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.038599 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.058342 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:45.079473 17349 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:45.079594 17349 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:45.080021 17349 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:45.081391 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:45.081521 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:45.081584 29404 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:45.081732 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:45.081735 17349 raft_consensus.cc:2726] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:45.104796 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.111172 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.208344 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.250994 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:45.262475 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.270743 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:45.282413 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.308050 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:45.313308 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:45.318785 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:45.349591 17349 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:45.349720 17349 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:45.350181 17349 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:45.351159 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:45.351279 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:45.351336 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:45.351538 17339 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:45.371403 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.391604 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.461709 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.472826 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:45.481519 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:45.533826 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:01:45.558538 17347 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:45.558652 17347 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:45.559034 17347 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:45.560041 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:45.560165 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:45.560221 31925 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:45.560351 17347 raft_consensus.cc:2726] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:45.572129 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:45.609042 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.624122 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:45.680646 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:45.685120 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.706122 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.724131 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:45.782615 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.785871 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.794430 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:45.808008 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.809847 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.820297 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.835474 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:45.836804 17338 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:45.836905 17338 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:45.837253 17338 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:45.838353 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:45.838464 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:45.838517 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:45.838635 17338 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:45.838661 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.883096 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:45.887084 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.915271 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:45.916440 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.918102 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:45.921602 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.926172 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:45.941402 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:45.948520 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.969089 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.007146 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.038434 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.060187 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:46.068907 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.084249 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:46.087517 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.159512 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:46.164974 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:46.189607 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.231215 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.234376 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.256445 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.269258 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.311599 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.333104 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:46.346104 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.388178 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.409919 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.421419 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:46.427856 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.444190 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.448375 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.449486 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.453889 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.455912 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.478709 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.494863 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.506309 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:01:46.530347 17349 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:46.530472 17349 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:46.531008 17349 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:46.531996 31990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:46.532140 31990 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:46.533116 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.533206 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:46.533288 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:46.533331 29404 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:46.533450 17349 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:46.538976 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.545835 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.552414 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.560431 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:46.574462 17352 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:46.574560 17352 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:46.575073 17352 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), efdbf7bdfe0b41bdadb1add053b09a7e (127.22.55.131:38105)
I20240502 14:01:46.575541 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" is_pre_election: true
I20240502 14:01:46.575696 29732 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:46.576432 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:46.576542 32188 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:46.576593 32188 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:46.576736 17352 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:46.577811 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.618132 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.628825 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.633482 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.665992 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:46.681123 17353 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:46.681236 17353 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:46.681771 17353 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:46.682032 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:46.682171 32254 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:01:46.682391 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:46.682763 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:46.682839 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:46.682994 17353 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:46.683511 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:46.688484 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.698367 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.715556 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:46.725428 17353 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:46.725554 17353 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:46.726058 17353 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:46.727772 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:46.727958 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:46.728021 29404 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:46.728183 17353 raft_consensus.cc:2726] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:46.728578 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:46.747601 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:46.751199 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.758596 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.807679 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.810736 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.812947 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:46.829273 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:01:46.829674 17349 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:46.829766 17349 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:46.830132 17349 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:46.831074 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:46.831192 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:46.831245 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:46.831398 17349 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:46.843042 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:46.848651 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.848872 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.850705 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:46.860569 17352 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:46.860654 17352 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:46.861011 17352 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:46.862229 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:46.862357 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:46.862411 32188 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:46.862556 17352 raft_consensus.cc:2726] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:46.865854 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.884866 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:46.910483 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.927428 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:46.962546 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:46.974097 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:47.007970 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.043521 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.094573 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.101147 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.135176 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.147150 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.149205 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.159375 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.203940 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:47.204990 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:01:47.220954 17352 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:47.221086 17352 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:47.221801 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:47.221932 29864 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
I20240502 14:01:47.222124 17352 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
W20240502 14:01:47.222757 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:47.222963 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:47.223019 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:47.223137 17352 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:47.223562 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:47.237722 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:47.242223 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.254607 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.305416 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.361115 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.371009 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:47.376413 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:01:47.377665 17338 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:47.377750 17338 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:47.378072 17338 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:47.379045 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:47.379173 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:47.379225 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:47.379369 17338 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:47.387970 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.395560 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:47.413313 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.423878 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.519644 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.591037 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.605923 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.655298 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:47.721661 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:47.725975 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.728910 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.752686 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.813887 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:47.815425 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.838683 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.871403 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.888283 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:47.897188 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:47.927657 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.965368 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:47.973970 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:47.993139 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:47.995311 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:47.996546 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.013489 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.013908 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.032456 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.104907 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.140866 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:48.244752 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 317 similar messages]
W20240502 14:01:48.254653 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.291131 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:48.297098 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:48.330190 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.330668 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 227 similar messages]
W20240502 14:01:48.337674 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.342653 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.348013 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.349998 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.357211 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.380928 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 246 similar messages]
W20240502 14:01:48.402781 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.413035 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.416213 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 208 similar messages]
W20240502 14:01:48.416779 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:01:48.420445 17349 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:48.420579 17349 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:48.420950 17349 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:48.421957 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:48.422065 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:48.422122 29404 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:48.422273 17349 raft_consensus.cc:2726] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:48.470986 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.499096 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.505352 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.522941 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.529546 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.530719 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.534739 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.544845 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.575568 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:48.603655 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.615128 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.619895 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:48.620501 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.622076 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.641001 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.644150 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:48.667939 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.695276 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.711349 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:48.727638 17362 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:48.727764 17362 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:48.728165 17362 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:48.729197 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:48.729310 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:48.729367 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:48.729496 17362 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:48.771163 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.811059 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.822755 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240502 14:01:48.823459 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.824028 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:48.828872 17362 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:48.828960 17362 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:48.829371 17362 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:48.830963 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:48.831077 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:48.831125 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:48.831318 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:48.832093 17362 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:48.834370 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.883486 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.891005 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:48.897796 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:48.914186 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.925930 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.942035 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.955366 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:48.970584 17360 raft_consensus.cc:482] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:48.970717 17360 raft_consensus.cc:504] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:48.971081 17360 leader_election.cc:288] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:48.971911 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:48.972246 29797 leader_election.cc:334] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:48.972299 29797 leader_election.cc:302] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:48.972419 17360 raft_consensus.cc:2726] T 0867c6a22c2c4efd9a9babff371cc8ba P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:48.980389 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:48.983799 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.008797 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.033311 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.034502 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:49.049527 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.068293 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.070462 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.081902 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.093631 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:49.110175 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.127213 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.138213 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.141765 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.147570 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.150627 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.153944 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:49.166563 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.171204 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.185967 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:49.227169 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:49.228508 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.245736 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.305856 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.324007 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:49.337657 17360 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:49.337770 17360 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:49.338279 17360 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:49.339416 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:49.339498 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:49.339547 29797 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:49.339666 17360 raft_consensus.cc:2726] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:49.339674 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:49.343946 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:49.346164 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.388883 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.401450 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:49.417984 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.431833 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:49.456390 17352 raft_consensus.cc:482] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:49.456502 17352 raft_consensus.cc:504] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } }
I20240502 14:01:49.456837 17352 leader_election.cc:288] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:49.457783 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:49.457897 32188 leader_election.cc:334] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:49.457955 32188 leader_election.cc:302] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:49.458120 17352 raft_consensus.cc:2726] T 7ede8318b945487cae52b2140358b2d9 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:49.490768 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.500941 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:49.506824 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.515358 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.539923 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.555642 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.575167 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.582548 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.609915 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.620466 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.637873 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.646678 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:49.651835 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:49.673184 17358 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:49.673318 17358 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:49.673692 17358 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:49.674616 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:49.674712 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:49.674765 29666 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:49.674906 17358 raft_consensus.cc:2726] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:49.722183 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.737679 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.781162 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:49.804498 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.807019 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:49.819389 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.883387 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.888680 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:49.916256 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:49.956460 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.040351 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:50.060282 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.079264 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.103583 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:50.110450 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.140887 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.146196 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:50.169364 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.238224 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.239617 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:50.288069 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.346662 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:50.357529 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:50.377693 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:50.382926 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.397689 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.407892 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:50.412583 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.438647 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.458083 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:50.470572 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.479532 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.480680 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:50.525990 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:50.539296 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:50.596316 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.602260 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.604043 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.654031 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.721406 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:50.752892 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.758838 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.759038 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.791923 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.813411 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.818434 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.820652 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.838263 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.860733 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:50.866190 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.873281 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:50.881994 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:50.906769 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:50.930259 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.972048 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.007352 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:51.016167 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:51.067409 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:51.106134 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:01:51.112488 17370 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:51.112602 17370 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:51.113011 17370 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:51.114123 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:51.114230 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:51.114295 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:51.114426 17370 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:51.114447 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:51.151268 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:51.157768 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:01:51.186370 17368 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:51.186487 17368 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:51.186817 17368 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:51.187961 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:51.188094 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:51.188148 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:51.188275 17368 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:51.192118 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.202250 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.208170 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:51.210806 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:51.239018 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:51.239013 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:51.244024 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:51.293051 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:51.309665 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.318405 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.332885 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.336598 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.377126 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:51.383069 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.385921 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.446353 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.467087 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:51.480332 17378 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:51.480449 17378 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:51.480897 17378 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:51.481194 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:51.481338 32254 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
W20240502 14:01:51.481853 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:51.481988 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:51.482035 29666 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:51.482169 17358 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:51.510807 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:51.531443 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.545169 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.561244 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.568856 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.585753 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:51.590972 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.593981 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.594780 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.599179 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.601514 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.612953 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.622625 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:51.625695 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:51.638779 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.652014 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:51.666399 17370 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:51.666507 17370 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:51.666942 17370 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:51.667891 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:51.668015 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:51.668076 31925 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:51.668202 17370 raft_consensus.cc:2726] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:51.677273 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.698648 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.711208 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.721375 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:51.725759 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.728122 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:51.738175 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:51.740607 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.741694 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:51.793627 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:51.838224 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.841482 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:51.847488 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.857772 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:51.870882 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.871029 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:51.901954 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.944388 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:51.947003 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:51.959053 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:51.960011 17379 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:51.960121 17379 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:51.960594 17379 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:51.961055 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:51.961194 32254 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aebe6d7a3f34bf7b2bbbfbef1077463 in term 1.
W20240502 14:01:51.961205 29799 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:51.961606 29797 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:51.961674 29797 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:51.961804 17379 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:51.978837 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.003388 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:52.065739 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:52.091933 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.093449 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.103145 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:52.111256 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:52.113507 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.128432 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.136075 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:52.136579 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:52.159495 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:52.177428 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:52.234997 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.235988 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:52.282897 17379 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:52.283020 17379 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:52.283397 17379 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:52.284292 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:52.284385 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:52.284417 29797 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:52.284530 17379 raft_consensus.cc:2726] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:52.327368 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.333596 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.457085 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.486047 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:52.497843 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:52.504287 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.545090 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.550462 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:52.572913 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:01:52.574205 17382 raft_consensus.cc:482] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:52.574316 17382 raft_consensus.cc:504] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
W20240502 14:01:52.574682 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:52.574716 17382 leader_election.cc:288] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:52.575629 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:52.575778 29797 leader_election.cc:334] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:52.575838 29797 leader_election.cc:302] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:52.575975 17382 raft_consensus.cc:2726] T fe523f0c364f447cb2a62b599d030cca P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:52.596279 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.600785 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.617539 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:52.668869 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:52.691155 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:52.719064 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:01:52.745069 17383 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:52.745203 17383 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:52.752000 17383 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), 269b366a09ff435ea8ec8dc01d0f9f6e (127.22.55.134:44137), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:52.752969 17379 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:52.753062 17379 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
W20240502 14:01:52.753270 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:52.753396 29404 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:52.753407 17379 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:52.753463 29404 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:52.753736 31992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" is_pre_election: true
I20240502 14:01:52.753873 31992 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
I20240502 14:01:52.754748 17383 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:52.755273 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:52.755374 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:52.755422 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:52.755543 17379 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:52.767972 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:52.779889 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.796491 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:52.884960 17387 raft_consensus.cc:482] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:52.885087 17387 raft_consensus.cc:504] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:52.885550 17387 leader_election.cc:288] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:52.887043 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:52.887153 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:52.887210 31925 leader_election.cc:302] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:01:52.887365 31925 leader_election.cc:334] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:52.887840 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:52.887881 17370 raft_consensus.cc:2726] T 09f77f3b6bf14a46b2e41c91125f6bd7 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:52.897912 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:52.900509 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:52.935057 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:52.945111 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:52.993430 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:53.005339 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.012600 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.051533 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.061596 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.063030 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.085691 17370 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:53.085814 17370 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:53.086197 17370 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:53.087582 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:53.087693 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:53.087750 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:53.087893 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:53.088024 17370 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:53.088320 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:01:53.108707 17370 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:53.108836 17370 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:53.109184 17370 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:53.110194 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:53.110312 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:53.110368 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:53.110497 17370 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:53.110670 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.116776 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.151294 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.163513 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.182019 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:53.203900 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.225924 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.246711 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 299 similar messages]
W20240502 14:01:53.254585 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.275051 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.283097 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:01:53.287047 17383 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:53.287150 17383 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:53.287559 17383 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:53.288398 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.288851 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:53.288954 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:53.289024 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:53.289177 17383 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:53.298913 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.330781 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.332168 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.375698 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 238 similar messages]
W20240502 14:01:53.376657 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.385776 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:53.390656 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.418442 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 248 similar messages]
W20240502 14:01:53.430397 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 216 similar messages]
W20240502 14:01:53.433048 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.439846 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.446717 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.479385 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:53.497951 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:53.503216 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.530175 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.577526 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.624173 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.625867 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:53.640853 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.663832 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:01:53.665246 17387 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:53.665341 17387 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:53.665668 17387 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:53.666707 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:53.666821 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:53.666896 31925 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:53.667049 17387 raft_consensus.cc:2726] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:53.681376 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:53.716678 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.722640 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.775633 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.793002 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.794873 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.823278 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.845420 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240502 14:01:53.860512 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.876874 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.896750 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.899976 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:01:53.924544 17382 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:53.924652 17382 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:53.924988 17382 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:53.925952 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:53.926035 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:53.926069 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:53.926195 17382 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:53.930416 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:53.935443 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:53.972728 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:53.996240 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:54.016045 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.016918 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:54.029600 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.032362 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.039847 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.052474 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.071035 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:54.081439 17387 raft_consensus.cc:482] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:54.081557 17387 raft_consensus.cc:504] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:54.081960 17387 leader_election.cc:288] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:54.081943 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.082896 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:54.083007 31925 leader_election.cc:334] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:54.083042 31925 leader_election.cc:302] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:54.083184 17387 raft_consensus.cc:2726] T 437ff61fd11046c5ac548d356e5e3402 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:54.094406 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.094815 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:54.107000 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:54.120501 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.123497 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.133041 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.139681 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.143030 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.153533 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.164641 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.168421 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.187775 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.209885 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:54.213408 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.213753 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.236881 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.243618 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.344192 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:54.348981 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.354493 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.363713 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.439922 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:54.443480 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.463907 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:54.468429 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:54.471822 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.478675 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:54.495699 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:54.522771 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.541878 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.545477 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.556058 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.564539 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.625677 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:01:54.680193 17397 raft_consensus.cc:482] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:54.680313 17397 raft_consensus.cc:504] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:54.680671 17397 leader_election.cc:288] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:54.681625 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:54.681738 32188 leader_election.cc:334] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:54.681792 32188 leader_election.cc:302] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:54.682010 17397 raft_consensus.cc:2726] T f787b154dfd44dfe9cf1ef2f996bfffb P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:54.696417 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.714710 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:54.726569 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.739514 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.746104 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.761165 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.765914 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:54.772831 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.787705 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.819137 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.821074 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:54.872020 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:54.894534 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:54.902173 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:54.950829 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:55.029870 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.037091 17392 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:55.037199 17392 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:55.037557 17392 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:55.039726 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:55.039834 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:55.039891 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:55.040024 17392 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:55.048408 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:55.052259 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.073446 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:55.083492 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.208895 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.212620 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:55.267915 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.280087 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.287874 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.309332 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.341248 17397 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:55.341374 17397 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:55.341884 17397 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:01:55.342231 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:55.342373 29471 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:55.342525 32189 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:01:55.343055 32188 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:55.343108 32188 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:55.343223 17397 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:55.346882 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:55.358079 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.400974 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:55.423465 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.432788 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:55.451928 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:55.464637 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:55.464849 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:55.503988 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:55.524042 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:55.531888 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.562790 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.562907 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.562996 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.567799 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.581156 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.600710 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.602979 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.610307 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.723798 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:55.755278 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:55.763900 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.764133 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:55.764168 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.780480 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.785744 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.795365 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.804263 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.912688 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:55.921525 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:55.938797 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:55.941170 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:55.955442 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:55.962383 17399 raft_consensus.cc:482] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:55.962510 17399 raft_consensus.cc:504] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:55.962962 17399 leader_election.cc:288] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:55.964320 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:55.964435 29404 leader_election.cc:334] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:55.964488 29404 leader_election.cc:302] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:55.964607 17399 raft_consensus.cc:2726] T d000011409c64f1a9fd6b5ad652d9a4a P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:55.966102 17402 raft_consensus.cc:482] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:55.966207 17402 raft_consensus.cc:504] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:01:55.967085 17402 leader_election.cc:288] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:55.967768 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:55.967893 29797 leader_election.cc:334] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:55.967944 29797 leader_election.cc:302] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:55.968075 17402 raft_consensus.cc:2726] T a113227860bc48078340ec76873be7e1 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:55.982748 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:56.032799 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.041646 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:56.057590 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:56.059970 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:56.077637 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:56.107340 17405 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:56.107556 17405 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
W20240502 14:01:56.109423 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:56.109530 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:56.109572 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:56.109691 17387 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:56.109943 17405 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:56.109979 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:56.110103 29471 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:56.115288 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.116156 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:56.139469 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.176396 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:56.194288 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:56.225090 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:56.230885 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:56.246083 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.258824 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.281582 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.311120 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:56.327363 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:56.332530 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:56.344520 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:56.353020 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:56.354898 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.394475 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:56.398212 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.405526 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:56.416005 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.417050 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.448421 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.464650 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.466822 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:56.471032 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.485996 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.512094 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.526796 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.556196 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.559284 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.601452 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.611151 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.648656 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:56.667673 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.668895 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.678367 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.684594 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.721665 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.728252 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:56.728627 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.747020 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.747046 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.760267 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:56.800941 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.803545 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:56.835815 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.856063 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.883793 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.894889 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:56.899664 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:56.937789 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:56.975194 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:56.994764 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.022464 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:57.058058 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.061748 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:01:57.068902 17394 raft_consensus.cc:482] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:57.069028 17394 raft_consensus.cc:504] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:57.069423 17394 leader_election.cc:288] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:57.070386 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.070894 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.070971 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:57.071038 29797 leader_election.cc:334] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:57.071089 29797 leader_election.cc:302] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:57.071233 17394 raft_consensus.cc:2726] T 995449d84e4b4d4692f8f225b5a44830 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:57.071872 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:57.108551 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.109889 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.110316 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:57.136438 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:57.151826 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:57.161810 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:57.216125 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:57.251509 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:57.275211 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:57.322206 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:57.331507 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:57.335953 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.339305 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.379981 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:57.397397 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.400856 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:01:57.475507 17412 raft_consensus.cc:482] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:57.475654 17412 raft_consensus.cc:504] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:57.476027 17412 leader_election.cc:288] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:57.476958 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:57.477063 29404 leader_election.cc:334] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:57.477118 29404 leader_election.cc:302] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:57.477280 17412 raft_consensus.cc:2726] T 6eae206e12b44b809d8b2fcc21611a28 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:01:57.491590 17413 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:57.491711 17413 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:01:57.492215 17413 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:01:57.492460 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:01:57.492599 32254 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
W20240502 14:01:57.493188 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:57.493297 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:57.493352 29666 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:57.493475 17413 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:57.497009 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:57.550205 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.571293 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:57.593875 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.603924 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.639636 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.649693 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:57.673074 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:01:57.718392 17405 raft_consensus.cc:482] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:01:57.718528 17405 raft_consensus.cc:504] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
W20240502 14:01:57.718623 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:57.719025 17405 leader_election.cc:288] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:57.719401 29471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdb9b4a6c07488f9bc2d15cba59214e" candidate_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d00d3a47c52478894cf8c66959ef7bd" is_pre_election: true
I20240502 14:01:57.719540 29471 raft_consensus.cc:2454] T fbdb9b4a6c07488f9bc2d15cba59214e P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269b366a09ff435ea8ec8dc01d0f9f6e in term 1.
W20240502 14:01:57.719913 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:57.720048 31925 leader_election.cc:334] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:57.720113 31925 leader_election.cc:302] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:57.720237 17405 raft_consensus.cc:2726] T fbdb9b4a6c07488f9bc2d15cba59214e P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:57.734166 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:57.742964 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.744391 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:57.763053 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:57.764001 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:57.776821 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.793756 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.794451 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:57.802739 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:57.827522 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:57.829026 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:57.838512 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.860255 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:57.953058 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.001641 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.056471 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.056790 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.057178 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.058161 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:58.083935 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.106750 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.114342 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.174255 17402 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:58.174378 17402 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:01:58.174686 17402 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:01:58.175868 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:58.175956 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:58.175989 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:58.176090 17402 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:58.192459 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:58.193334 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.248549 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:58.256603 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.275498 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 304 similar messages]
W20240502 14:01:58.287009 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.292322 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:58.302311 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.347815 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.360073 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.362347 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:58.384800 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.390540 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 230 similar messages]
W20240502 14:01:58.404229 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.407553 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.413492 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.419278 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 248 similar messages]
W20240502 14:01:58.423190 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.431281 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 213 similar messages]
W20240502 14:01:58.443056 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:01:58.459218 17405 raft_consensus.cc:482] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:58.459326 17405 raft_consensus.cc:504] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:58.459687 17405 leader_election.cc:288] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:58.460901 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:58.461004 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:01:58.461061 31925 leader_election.cc:302] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
W20240502 14:01:58.461192 31925 leader_election.cc:334] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:58.461203 17405 raft_consensus.cc:2726] T 7ece4fc38b254afeae58936444fe047d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:58.513648 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.537755 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.568300 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.581869 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:01:58.660915 17418 raft_consensus.cc:482] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:58.661047 17418 raft_consensus.cc:504] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:01:58.661463 17418 leader_election.cc:288] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:01:58.662402 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:58.662523 31925 leader_election.cc:334] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:01:58.662565 31925 leader_election.cc:302] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:01:58.662683 17418 raft_consensus.cc:2726] T 8d2ab1ded2c241038d5685678aa819a8 P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:58.680820 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.682493 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:58.752310 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.771204 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.788944 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:58.794293 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.799276 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.802245 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:58.831902 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:58.844233 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.851370 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.858460 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.865900 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:58.881917 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.885167 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.886160 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:58.900246 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240502 14:01:58.905997 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.909756 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:58.915272 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.930091 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.946053 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.950574 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.954384 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.957792 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.972389 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:58.973368 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.025794 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.033576 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.039543 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.075145 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.078711 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:59.089221 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.101204 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.129176 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:59.163661 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:59.231734 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.231886 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.236145 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:59.236258 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.250933 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:59.251029 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:59.253955 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.256104 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.260178 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.293588 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.306816 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.308001 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:59.308936 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:59.316219 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:59.327530 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.330041 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.381134 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.487926 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:59.499526 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.504169 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:59.568672 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:59.601234 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.615612 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.616425 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:59.640264 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:59.656765 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:59.673936 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.674415 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:01:59.675588 17397 raft_consensus.cc:482] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
I20240502 14:01:59.675673 17397 raft_consensus.cc:504] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:01:59.676051 17397 leader_election.cc:288] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823)
I20240502 14:01:59.676537 29864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31df5c53f5004ef399a1bd3458f57573" candidate_uuid: "8973f2527acb48f5ad36589652de73e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" is_pre_election: true
I20240502 14:01:59.676668 29864 raft_consensus.cc:2454] T 31df5c53f5004ef399a1bd3458f57573 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8973f2527acb48f5ad36589652de73e1 in term 1.
W20240502 14:01:59.676780 32190 leader_election.cc:341] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d00d3a47c52478894cf8c66959ef7bd (127.22.55.129:44823): Not found: Tablet not found: 31df5c53f5004ef399a1bd3458f57573
W20240502 14:01:59.676959 32188 leader_election.cc:334] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:59.677031 32188 leader_election.cc:302] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, 8973f2527acb48f5ad36589652de73e1; no voters: 1d00d3a47c52478894cf8c66959ef7bd, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:59.677161 17397 raft_consensus.cc:2726] T 31df5c53f5004ef399a1bd3458f57573 P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:59.686612 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:01:59.750344 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.756294 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:59.760088 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:59.789592 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.814083 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.826361 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:01:59.884268 17402 raft_consensus.cc:482] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:01:59.884392 17402 raft_consensus.cc:504] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } }
I20240502 14:01:59.884765 17402 leader_election.cc:288] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
W20240502 14:01:59.885670 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:59.885788 29797 leader_election.cc:334] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:01:59.885848 29797 leader_election.cc:302] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:01:59.885986 17402 raft_consensus.cc:2726] T bd91ed1ae1354a84a0ff668efdbc03d9 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:01:59.897868 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:59.912168 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:01:59.927814 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.956025 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:01:59.959812 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:59.988915 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:59.994251 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.005281 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:00.040115 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.051911 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.086954 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.143030 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:00.162472 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:00.164431 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.173261 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.199680 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:00.223203 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.229938 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.289013 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.292847 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.375293 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.412405 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.417039 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.464151 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:02:00.473341 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.500244 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.503207 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:00.518265 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.520433 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.530547 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:00.559252 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.598820 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.615015 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.632884 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:00.634672 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.668902 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:00.682629 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.712533 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.723789 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.727113 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.802513 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.824090 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.857120 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.862143 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:00.863122 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.878059 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.894182 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:00.919138 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:00.981868 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:00.991487 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.008481 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.010869 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.017232 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:01.046603 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:01.051836 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.084379 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.096118 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:01.096791 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.157240 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.166007 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.202817 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:01.215059 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:01.235556 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:01.250481 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:01.255355 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.268882 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:01.279109 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.305892 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.309688 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.313416 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.327250 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.327280 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.362394 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:01.369516 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.382920 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:01.403419 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.409363 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.447839 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.477327 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.484120 29404 consensus_peers.cc:485] T fad144c461044c299ea905832908666a P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.489133 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.490767 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:01.508189 31925 consensus_peers.cc:485] T 4a2ab852e31d4181a8597b9047a08554 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:01.514676 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.534533 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:01.555739 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:01.611124 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:01.634158 17402 raft_consensus.cc:482] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:02:01.634281 17402 raft_consensus.cc:504] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:02:01.634719 17402 leader_election.cc:288] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:02:01.635579 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:01.636114 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:01.636365 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:02:01.636420 29797 leader_election.cc:302] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:02:01.636556 29797 leader_election.cc:334] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:02:01.636561 17402 raft_consensus.cc:2726] T 3062c7a80b284f6c8cb28f8c53c9c0a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:01.638253 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:01.664153 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.676733 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.679697 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.699949 31925 consensus_peers.cc:485] T 13d5181b92f946b99c26735bb05b5974 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.702549 29797 consensus_peers.cc:485] T 8e41bff5f1484f008851e76cc095eb91 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.734107 29404 consensus_peers.cc:485] T aeaf1db0298741fa813660a8e38d01f9 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.760798 32188 consensus_peers.cc:485] T 39db58bc129046bbbefea02b3eee4243 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:01.773597 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.797207 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.805516 31925 consensus_peers.cc:485] T 1fc14b4bd42841df91972e0c5d02faa4 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.862296 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.869271 32188 consensus_peers.cc:485] T 93e9d6f5e9b84267afa5e019f6bb8c08 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:02:01.873741 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.875473 29797 consensus_peers.cc:485] T 9790e7ee6d8945bf85f129ef3f9900ca P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:01.928645 29666 consensus_peers.cc:485] T 4c08999e4e7f4d0bb6a03199ee676f76 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:01.946230 29666 consensus_peers.cc:485] T 764d8861015246c6a37dfdbd1e3f48e2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.968420 29404 consensus_peers.cc:485] T a7cc49f0227d491ba7a8df281856b8f8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:01.975793 32188 consensus_peers.cc:485] T 0d2acfdaac334736a065a7850e8293ba P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.009310 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.021965 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:02.034806 31925 consensus_peers.cc:485] T 3e309595cfaf4bf78e6fe9594018696d P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.034929 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:02.043066 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.055500 32188 consensus_peers.cc:485] T 19d48f827a6243448d344ee44a54482d P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.068130 29666 consensus_peers.cc:485] T f8b939db0e5b464dbcecbf8e18cccadf P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:02.087199 17418 raft_consensus.cc:482] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:02:02.087321 17418 raft_consensus.cc:504] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:02:02.087764 17418 leader_election.cc:288] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:02:02.088797 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:02.088907 31925 leader_election.cc:334] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:02:02.088977 31925 leader_election.cc:302] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:02:02.089146 17418 raft_consensus.cc:2726] T 8a7bfc5231a542dda0a51aee4543299d P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:02.123258 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:02.132004 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.155689 29666 consensus_peers.cc:485] T 3d2ed5a4845b4bd78b19af6380c77982 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.201645 32188 consensus_peers.cc:485] T ed0c8871730c4e2ba9f1320381159331 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:02.208706 32188 consensus_peers.cc:485] T 22543bedc3f043bba1d05b675a8f3c75 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:02.212679 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.213887 29797 consensus_peers.cc:485] T 925eac6300a44f17bbbf3fe742bc2a26 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.214186 29666 consensus_peers.cc:485] T 209e6922777b49ddb87a3dadf65d651a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.219388 31925 consensus_peers.cc:485] T 00d11f638cfa4b53b532c9084fa45610 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.243374 31925 consensus_peers.cc:485] T f5f316f08f5041a4907ad768c28ae30f P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.252589 29797 consensus_peers.cc:485] T b415bd7b51fc4e4a8c6fa6083d989bfe P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:02.261431 31925 consensus_peers.cc:485] T 3afd58ca519e4bc8a6ab00c37abf9f1a P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.271754 32188 consensus_peers.cc:485] T 70367c7c718d4d9195f0af7e69bdf2ec P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:02.302790 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.307577 29797 consensus_peers.cc:485] T e4a84488994c47ba9436ea7161d9b2e3 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.341377 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.408356 31925 consensus_peers.cc:485] T 9e2dc222951548708862c3adf205d0ef P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:02.476366 17430 raft_consensus.cc:482] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f6b3fa3988ae4af9ba8cc5ffb79d5bfb)
W20240502 14:02:02.476467 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:02:02.476508 17430 raft_consensus.cc:504] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } }
I20240502 14:02:02.477175 17430 leader_election.cc:288] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915)
I20240502 14:02:02.477361 32254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23fc8a56f054a5b8ec0272ad99a93ea" candidate_uuid: "1d00d3a47c52478894cf8c66959ef7bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:02:02.477501 32254 raft_consensus.cc:2454] T b23fc8a56f054a5b8ec0272ad99a93ea P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d00d3a47c52478894cf8c66959ef7bd in term 1.
W20240502 14:02:02.477900 29405 leader_election.cc:341] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4aebe6d7a3f34bf7b2bbbfbef1077463 (127.22.55.132:44915): Not found: Tablet not found: b23fc8a56f054a5b8ec0272ad99a93ea
W20240502 14:02:02.478431 29404 leader_election.cc:334] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:02:02.478492 29404 leader_election.cc:302] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd, 8973f2527acb48f5ad36589652de73e1; no voters: 4aebe6d7a3f34bf7b2bbbfbef1077463, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:02:02.478616 17430 raft_consensus.cc:2726] T b23fc8a56f054a5b8ec0272ad99a93ea P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:02.484587 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:02.550175 31925 consensus_peers.cc:485] T 5e1e00b442df441da78bbf2fcda5be73 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:02.581677 17426 raft_consensus.cc:482] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:02:02.581794 17426 raft_consensus.cc:504] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269b366a09ff435ea8ec8dc01d0f9f6e" member_type: VOTER last_known_addr { host: "127.22.55.134" port: 44137 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:02:02.582202 17426 leader_election.cc:288] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:02:02.583115 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:02.583205 31925 leader_election.cc:334] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:02:02.583259 31925 leader_election.cc:302] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 269b366a09ff435ea8ec8dc01d0f9f6e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:02:02.583397 17426 raft_consensus.cc:2726] T 8579d75d058e463b85fa5c982d74bf3f P 269b366a09ff435ea8ec8dc01d0f9f6e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:02.592780 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:02.619516 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.626940 29666 consensus_peers.cc:485] T 09ce1f1d33dd4beca86a484a6f498175 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.628701 32188 consensus_peers.cc:485] T 54f01d68468f4afca23ca423748ddb58 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.641079 31925 consensus_peers.cc:485] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:02.685508 31925 consensus_peers.cc:485] T aeab77cba6cf431e89141d4154c41a9b P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:02:02.705039 29404 consensus_peers.cc:485] T 5ea779bcca6749e196a4ed3ad8772a19 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.733574 32188 consensus_peers.cc:485] T 4e8de7efb7024d0290b0e313e7b3d5fd P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.753860 32188 consensus_peers.cc:485] T 1c91754a477542e5abea67a09971d111 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.790994 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.809653 29797 consensus_peers.cc:485] T bf3e82b3dc394648a13eafe4ec832fb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:02.829913 32188 consensus_peers.cc:485] T 6df051496f954569983892bfe828f653 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:02.848002 17430 raft_consensus.cc:482] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:02:02.848125 17430 raft_consensus.cc:504] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } }
I20240502 14:02:02.848496 17430 leader_election.cc:288] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:02:02.849557 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:02.849692 29404 leader_election.cc:334] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:02:02.849742 29404 leader_election.cc:302] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:02:02.849906 17430 raft_consensus.cc:2726] T 01b3fb8ad72148328d8f9d5750c0bd14 P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:02.872910 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:02.880057 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:02.921864 29797 consensus_peers.cc:485] T e9f48a525d8c4a0783f0450aa6ec7cb0 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:02.926168 29666 consensus_peers.cc:485] T 25d043c246e94f3090295583133ca0cd P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:02.983872 32188 consensus_peers.cc:485] T 2868541b54c447219db948c92b9a132b P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:02.988312 29666 consensus_peers.cc:485] T ef09c96ab4314611a0d51383ad31249b P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.010229 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:03.015345 29666 consensus_peers.cc:485] T 1089cd67cbd24551836ccb702ff44d4a P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.064711 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:03.136756 29797 consensus_peers.cc:485] T 1fefaee1ab4c4a77aa7ed79d6c2e8894 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.151177 29404 consensus_peers.cc:485] T 266aa0d4f078466a820bd48e60d044aa P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.157413 32188 consensus_peers.cc:485] T 1ea3465ad91540cea8906bcc406efd07 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.161644 29666 consensus_peers.cc:485] T be8afc0891fb43fa9ffc863ad2bc90d2 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.187897 29404 consensus_peers.cc:485] T f716493d95f84740beb85c3bf4021ad0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.196259 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.213358 32188 consensus_peers.cc:485] T b19374a33aaf41069e98fd035d2b9f00 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.215538 31925 consensus_peers.cc:485] T abcdb2e98b804235b1196a434113c861 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.225626 32188 consensus_peers.cc:485] T 7c83270b40d541939325d058bcf807e4 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:03.279512 17384 raft_consensus.cc:482] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:02:03.279626 17384 raft_consensus.cc:504] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:02:03.280012 17384 leader_election.cc:288] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:02:03.280395 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 302 similar messages]
W20240502 14:02:03.281013 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:03.281127 32188 leader_election.cc:334] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:02:03.281181 32188 leader_election.cc:302] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1; no voters: d6e2e2c10def4331bec17bd110c9b423, f0e01c4f581843e9ad4ced97f0072dc4
I20240502 14:02:03.281327 17384 raft_consensus.cc:2726] T b0fa578318964569b2e10d7f858acd4d P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:03.309805 29797 consensus_peers.cc:485] T 82cda992fe784d9f9015e6a3d431f1cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:03.315027 31925 consensus_peers.cc:485] T 5733940f27f345d8a314a9e51d2a455e P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:03.318684 29666 consensus_peers.cc:485] T 27e9d8ef4f754df3817cc4a981549e02 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.363739 29404 consensus_peers.cc:485] T ea564959c91a43cb9cac2f58cece4de8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.372782 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.381237 29797 consensus_peers.cc:485] T 9bdf925a9c4d456fb3b4ef448a2154c6 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:03.387498 17436 raft_consensus.cc:482] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:02:03.387636 17436 raft_consensus.cc:504] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "4aebe6d7a3f34bf7b2bbbfbef1077463" member_type: VOTER last_known_addr { host: "127.22.55.132" port: 44915 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:02:03.388051 17436 leader_election.cc:288] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:02:03.388499 32188 consensus_peers.cc:485] T 3a34a1381cf84af6b560101d95447e6d P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.389200 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:03.389303 29797 leader_election.cc:334] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:02:03.389367 29797 leader_election.cc:302] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aebe6d7a3f34bf7b2bbbfbef1077463; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:02:03.389747 17436 raft_consensus.cc:2726] T 75347e9c867547efaf78355aec6dfb40 P 4aebe6d7a3f34bf7b2bbbfbef1077463 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:02:03.390264 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:03.402810 31925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 230 similar messages]
W20240502 14:02:03.431448 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111) [suppressed 248 similar messages]
W20240502 14:02:03.432106 29797 consensus_peers.cc:485] T 4e69badccfbc416f92d982cf394ea0cd P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.432904 29666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111) [suppressed 207 similar messages]
W20240502 14:02:03.449244 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:03.476056 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:03.540953 29797 consensus_peers.cc:485] T 7650009dd8954f11ab2ecafef765692c P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:03.543215 29666 consensus_peers.cc:485] T 891851115e2e4d9db5bf3a0013b0edba P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.605204 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.610942 29797 consensus_peers.cc:485] T 198b0a675f184a268bf714cf778408a4 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.625520 32188 consensus_peers.cc:485] T 8151ac615c84417cab1b210a95a0f32c P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.632301 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.648461 29404 consensus_peers.cc:485] T 541806c11f074316a5342050427f4849 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.652074 29404 consensus_peers.cc:485] T f28e9f5068aa48b5a87841bdf720d9e0 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.656363 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:03.657670 29797 consensus_peers.cc:485] T 631725c869674c45ac79c3f05cee8356 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.663388 29404 consensus_peers.cc:485] T 8eb9e9de44a046ffb4594755f44f15b8 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:03.675731 17442 raft_consensus.cc:482] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:02:03.675843 17442 raft_consensus.cc:504] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } } peers { permanent_uuid: "1d00d3a47c52478894cf8c66959ef7bd" member_type: VOTER last_known_addr { host: "127.22.55.129" port: 44823 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } }
I20240502 14:02:03.676594 17442 leader_election.cc:288] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647)
W20240502 14:02:03.678267 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:03.678351 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:02:03.678383 29404 leader_election.cc:302] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1d00d3a47c52478894cf8c66959ef7bd; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
W20240502 14:02:03.678495 29404 leader_election.cc:334] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
I20240502 14:02:03.678506 17442 raft_consensus.cc:2726] T 253ea421888249d180928ba0aef28ddd P 1d00d3a47c52478894cf8c66959ef7bd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:03.681273 31925 consensus_peers.cc:485] T 0ec7b45e60d140d4a6e937307370fdc9 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:03.687320 29666 consensus_peers.cc:485] T 93e77c40b5674865ba6ccce626545ec7 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.704596 29666 consensus_peers.cc:485] T 444e705b42e7423b9ee1d187ea1de308 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.709612 31925 consensus_peers.cc:485] T d40f25aa6bcc46fca8cd67a6fd4b86b6 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.714987 29666 consensus_peers.cc:485] T 962095fe12324c90ae98a69a151438a6 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.734434 29797 consensus_peers.cc:485] T 59f4faff46704ca1837edd139596d9e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:03.789392 29666 consensus_peers.cc:485] T 463dddd3473a42a68b488e1c465e997f P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.797355 29404 consensus_peers.cc:462] T 6f1a1a1592e84b3898a06d4360bac846 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:03.851477 32188 consensus_peers.cc:462] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:03.853719 32188 consensus_peers.cc:485] T acdcc4d7e5754103a08d76bdd714a68e P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.870393 32188 consensus_peers.cc:485] T fe66bde51c1d4edfa739f6b9a833b129 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.872439 31925 consensus_peers.cc:485] T 1203440714974cb5981e67c2856fb0c7 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.876047 29797 consensus_peers.cc:485] T e78dc2ec29ad408e9665da00778743e2 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.881373 29404 consensus_peers.cc:485] T b2c754e60c1d4f6f95a807e9644bd21f P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:03.884037 17437 raft_consensus.cc:482] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f0e01c4f581843e9ad4ced97f0072dc4)
I20240502 14:02:03.884137 17437 raft_consensus.cc:504] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8973f2527acb48f5ad36589652de73e1" member_type: VOTER last_known_addr { host: "127.22.55.136" port: 44201 } } peers { permanent_uuid: "f0e01c4f581843e9ad4ced97f0072dc4" member_type: VOTER last_known_addr { host: "127.22.55.133" port: 39647 } } peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } }
I20240502 14:02:03.884559 17437 leader_election.cc:288] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8973f2527acb48f5ad36589652de73e1 (127.22.55.136:44201), f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647), f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995)
I20240502 14:02:03.884768 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab1144457d34c25897c5b7a691409fe" candidate_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8973f2527acb48f5ad36589652de73e1" is_pre_election: true
I20240502 14:02:03.884899 32255 raft_consensus.cc:2454] T eab1144457d34c25897c5b7a691409fe P 8973f2527acb48f5ad36589652de73e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efdbf7bdfe0b41bdadb1add053b09a7e in term 1.
W20240502 14:02:03.885555 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111)
W20240502 14:02:03.885666 29666 leader_election.cc:334] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
I20240502 14:02:03.885716 29666 leader_election.cc:302] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8973f2527acb48f5ad36589652de73e1, efdbf7bdfe0b41bdadb1add053b09a7e; no voters: f0e01c4f581843e9ad4ced97f0072dc4, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:02:03.885849 17437 raft_consensus.cc:2726] T eab1144457d34c25897c5b7a691409fe P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:03.933094 31925 consensus_peers.cc:485] T 768df43846d94951a3a48f6f626bb473 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.940359 32188 consensus_peers.cc:485] T a8e5364d50da4cfb8fbbf21913324bc6 P 8973f2527acb48f5ad36589652de73e1 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.946161 29404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111) [suppressed 169 similar messages]
I20240502 14:02:03.956411 17437 raft_consensus.cc:482] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d6e2e2c10def4331bec17bd110c9b423)
I20240502 14:02:03.956527 17437 raft_consensus.cc:504] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b3fa3988ae4af9ba8cc5ffb79d5bfb" member_type: VOTER last_known_addr { host: "127.22.55.135" port: 40995 } } peers { permanent_uuid: "efdbf7bdfe0b41bdadb1add053b09a7e" member_type: VOTER last_known_addr { host: "127.22.55.131" port: 38105 } } peers { permanent_uuid: "d6e2e2c10def4331bec17bd110c9b423" member_type: VOTER last_known_addr { host: "127.22.55.130" port: 40041 } }
I20240502 14:02:03.956907 17437 leader_election.cc:288] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995), d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041)
W20240502 14:02:03.957345 31925 consensus_peers.cc:485] T e574c148957f44bf9c2935e955de99aa P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:03.957878 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111)
W20240502 14:02:03.958269 29666 leader_election.cc:334] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
I20240502 14:02:03.958331 29666 leader_election.cc:302] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efdbf7bdfe0b41bdadb1add053b09a7e; no voters: d6e2e2c10def4331bec17bd110c9b423, f6b3fa3988ae4af9ba8cc5ffb79d5bfb
I20240502 14:02:03.958460 17437 raft_consensus.cc:2726] T 21a2c1b2947c47289ace8cf41f2fbb4d P efdbf7bdfe0b41bdadb1add053b09a7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:02:03.961594 29666 consensus_peers.cc:485] T df1c6786a80e40a98130a5d9057e2877 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.963716 32188 consensus_peers.cc:485] T 835183257412496b90defa7ecaa0ff13 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.971685 32188 consensus_peers.cc:485] T 2a41450335344abe9d261d6a00674293 P 8973f2527acb48f5ad36589652de73e1 -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:03.983979 29666 consensus_peers.cc:485] T 282d8021fee14b309b4a357f370aa334 P efdbf7bdfe0b41bdadb1add053b09a7e -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:04.004928 31925 consensus_peers.cc:485] T 70e59a9a97d9411e963d470b235706bd P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb (127.22.55.135:40995): Couldn't send request to peer f6b3fa3988ae4af9ba8cc5ffb79d5bfb. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.135:40995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:04.012403 31925 consensus_peers.cc:462] T 65229e977b8543639fb27bbbc3b5e797 P 269b366a09ff435ea8ec8dc01d0f9f6e -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111)
W20240502 14:02:04.046590 29797 consensus_peers.cc:485] T 14ea89aafe48423491f90865328ced42 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:04.067914 29797 consensus_peers.cc:485] T a0001d3f124c42eba6129836209a41fa P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:04.087436 32188 consensus_peers.cc:485] T 04f0f9efcbb544d7b8292158bf769426 P 8973f2527acb48f5ad36589652de73e1 -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn't send request to peer d6e2e2c10def4331bec17bd110c9b423. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.130:40041: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:04.091079 29797 consensus_peers.cc:485] T a4a635382b94471088de77b6c431d334 P 4aebe6d7a3f34bf7b2bbbfbef1077463 -> Peer f0e01c4f581843e9ad4ced97f0072dc4 (127.22.55.133:39647): Couldn't send request to peer f0e01c4f581843e9ad4ced97f0072dc4. Status: Network error: Client connection negotiation failed: client connection to 127.22.55.133:39647: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:04.092664 29404 consensus_peers.cc:485] T 1b37defc84d34af08f5744bf60a038c5 P 1d00d3a47c52478894cf8c66959ef7bd -> Peer d6e2e2c10def4331bec17bd110c9b423 (127.22.55.130:40041): Couldn'